All American Airlines flights grounded by computer outage

[sn26567]

American Airlines is experiencing a reservation system outage and has grounded all its flights in a "system-wide ground delay" until 5:00 pm ET (2100 GMT) as it works to resolve a reservation system outage.

Earlier, the FAA said that all American Airline flights were grounded at the carrier's request. The system outage affects its reservation and booking tool.

The flight delays are potentially affecting over 100,000 people affected. American Airlines runs more than 3,300 flights daily.

Other carriers have also had problems with system outages. Last year, computer outages at United Continental stranded passengers at airports around the country.

[Flanker2]

What a bunch of amateurs.

Maybe it's linked to the terror threat? After all it's easier to say "our reservation system broke down" than "we have reasons to believe that there is an active threat on American Airlines aircraft".

[jan_olieslagers]

Are you as knowledgeable about IT as about airline management? If so: do go for it, AA will be glad to welcome your genius and give them 100% reassurance this will NEVER AGAIN happen to them.

To those in the real world: shit happens - my sympathy to the IT'ers who must be under immense pressure to get things going again. Likely enough, they managed that, by now.

[Flanker2]

Dear jan, I appreciate your point of view but if this is really an IT problem, it's a big cock-up by AA.

Everybody knows that whenever an entire operation is supported by an IT system, you need to have a plan B.
You need to have back-up systems that would allow minimal functionalities and provide information on bookings for the next 7 days.

Back in the days, each station would print early in the morning before the first flights, a list of all the departing passengers for the same day and the following day, so that in case of this kind of things, there would be a plan B. There would have been delays as each passenger would have to be checked in manually, but it's still much better than what happened today.

Operating without a plan B is the perfect example of an amateur operation, regardless of any technical IT matters. Isn't it common sense?

[laurents]

I can confirm it is an IT issue. I'm due to fly with AA tomorrow and could not Check-in online. Eventhough I'm only flying tomorrow it shows me that my flight has already departed. Let's hope they can fix it quickly.

[Acid-drop]

Are you as knowledgeable about IT as about airline management? If so: do go for it, AA will be glad to welcome your genius and give them 100% reassurance this will NEVER AGAIN happen to them.

To those in the real world: shit happens - my sympathy to the IT'ers who must be under immense pressure to get things going again. Likely enough, they managed that, by now.

I am an IT'er.
and from my experience, in 99% of the case, a "computer issue" is just a stupid excuse to not say the truth.

[BrightCedars]

I'm in large scale IT myself, and to be honest, the likelyhood of having a single or even multiple IT failure grounding a carrier the size of AA has got to be inexistent. I vote for the cover up theory.

[sn26567]

All in all, 800 flights were cancelled. Expensive computer glitch!

[cnc]

I'm in large scale IT myself, and to be honest, the likelyhood of having a single or even multiple IT failure grounding a carrier the size of AA has got to be inexistent. I vote for the cover up theory.

why is that? all in all i find aviation DCS systems rather fragile

[Tomskii]

Dear jan, I appreciate your point of view but if this is really an IT problem, it's a big cock-up by AA.

Everybody knows that whenever an entire operation is supported by an IT system, you need to have a plan B.
You need to have back-up systems that would allow minimal functionalities and provide information on bookings for the next 7 days.

Back in the days, each station would print early in the morning before the first flights, a list of all the departing passengers for the same day and the following day, so that in case of this kind of things, there would be a plan B. There would have been delays as each passenger would have to be checked in manually, but it's still much better than what happened today.

Operating without a plan B is the perfect example of an amateur operation, regardless of any technical IT matters. Isn't it common sense?

If you would know what it takes to set up such systems and how many big companies have server outages every day: "ik zou een beetje dimmen als ik u was"

When one isn't familliar with how a server works and what it is, just refrain yourself from commenting before looking stupid.

@BrightCedars: Yeah that's why each day thousand of harddisks in servers of eg Facebook crash, the only reason we do not notice it, is because they have backups all over the world so you never lose data.

I would suppose AA's data centers do not have such a large amount of backups to go on. Secondly it isn't the first time an airline's booking and reservation system goes down...

[Acid-drop]

If you would know what it takes to set up such systems and how many big companies have server outages every day

Yes and nothing ever get noticed. Not even 10 seconds.

@BrightCedars: Yeah that's why each day thousand of harddisks in servers of eg Facebook crash, the only reason we do not notice it, is because they have backups all over the world so you never lose data.

There are different kind of IT jobs. There are those who makes the system, and those who spend their life making sure the system is doubled and that things never fail.
The whole system is designed to not fail. And even if it does, how can you explain that something that was working yesterday doesnt have backups and can't be restored in one hour.
It's just not possible. Not even worth to talk about this as an option.

[jan_olieslagers]

Some good replies and some less good.
Flanker2 has a point: everything system can and occasionally does go down, in aviation like in IT like anywhere; it is wise to prepare against such failure by having an IT-lees alternative. Whether this is still feasible in today's airline operations is unclear to me, but it does seem very bad to make one's operations, one's prime mode and reason of existence, totally dependent on IT that is less than perfect.

Acid-drop has a point too: "IT failure" is a handy excuse when managers don't wish to give out the real story. But OTOH there are cases where the trouble is _really_ due to IT unavailability.

Tomskii is IMHO opinion mistaken in the belief that the breaking of harddisks (which IS a common occurence indeed!) is remedied by restoring files from backup: in all but the least critical servers, most components are in failsafe configurations. Harddisks are in RAID configurations, so that one disk can fail without compromising operations. The defective disk can be swapped, and the configuration brought back to optimal, without any disruption. Availability is further increased by the use of hot-spare disks, that are in the disk array for the sole purpose of taking over the role of a disk gone defective.
Backups are important, and often enough a file does need to be restored (or even a whole set of files, or a dir hierarchy); but usually it is because some idiot wiped or overwrote some data, either her/his own or someone else's.
(power supplies are usually in N+1 or even N+2 redundant configs too, and may me swapped without interrupting service. IP and storage networks have multiple datalinks, also for redundancy)

But as you may have guessed I am in (big) IT too, and I know from hard and shameful experience what it feels like to commit a stupid typo that sets off a whole chain of disruptions. Shit happens, to IT-ers as much as to pilots - indeed I think the two professions are much alike, though I only know piloting as an amateur.

[Flanker2]

I've come up with a rough estimate that one day of total shut down causes to the operations of American Airlines. 18 billion dollars per year divided by 365 days... well, maybe a bit less but still

[Tomskii]

Some good replies and some less good.
Flanker2 has a point: everything system can and occasionally does go down, in aviation like in IT like anywhere; it is wise to prepare against such failure by having an IT-lees alternative. Whether this is still feasible in today's airline operations is unclear to me, but it does seem very bad to make one's operations, one's prime mode and reason of existence, totally dependent on IT that is less than perfect.

Acid-drop has a point too: "IT failure" is a handy excuse when managers don't wish to give out the real story. But OTOH there are cases where the trouble is _really_ due to IT unavailability.

Tomskii is IMHO opinion mistaken in the belief that the breaking of harddisks (which IS a common occurence indeed!) is remedied by restoring files from backup: in all but the least critical servers, most components are in failsafe configurations. Harddisks are in RAID configurations, so that one disk can fail without compromising operations. The defective disk can be swapped, and the configuration brought back to optimal, without any disruption. Availability is further increased by the use of hot-spare disks, that are in the disk array for the sole purpose of taking over the role of a disk gone defective.
Backups are important, and often enough a file does need to be restored (or even a whole set of files, or a dir hierarchy); but usually it is because some idiot wiped or overwrote some data, either her/his own or someone else's.
(power supplies are usually in N+1 or even N+2 redundant configs too, and may me swapped without interrupting service. IP and storage networks have multiple datalinks, also for redundancy)

But as you may have guessed I am in (big) IT too, and I know from hard and shameful experience what it feels like to commit a stupid typo that sets off a whole chain of disruptions. Shit happens, to IT-ers as much as to pilots - indeed I think the two professions are much alike, though I only know piloting as an amateur.

Thanks Jan,

The harddisks RAID config was indeed what I meant on top of the fact that for instance fb has backups on several locations as well so that if the rare case of a total crash of one datacenter nobody would notice anything (I am stating this correctly right?)

[jan_olieslagers]

It seems you are still confused with the word "backup". Backups are made for the purpose of restoring, i.e. making something available again after an unavailability. As I already stated, this usually follows from (minor) human error. In extreme stages of Disaster Recovery, a complete datacentre may be restored from backup - but the operation is likely to take days, if not weeks.

In a well done architecture, data is multiplicated over several datacentres, so that data will remain solid, and available, even if a complete datacentre fails. One well-known Belgian bank has such data replication between its data centres in Mechelen and Leuven, though that is too far apart to be really good - they had to spend a fair deal of money on extra measures to ensure data integrity. Another (ex-)Belgian bank has two sites in Brussels, the fibre optic cables reportedly tucked away in auxiliary cable ducts of the metro.

Of course the cost of such massive data protection is huge, only affordable for the real big spenders. This explains the success of third-party infrastructure operators ( I work for such an organisation) and the recent myth of the cloud.

[Passenger]

Belgium Great Country ! In the U.S., nobody suggested that it was a cover up. But fortunately for them, the Belgians did:

What a bunch of amateurs. Maybe it's linked to the terror threat? After all it's easier to say "our reservation system broke down" than "we have reasons to believe that there is an active threat on American Airlines aircraft".

I'm in large scale IT myself, and to be honest, the likelyhood of having a single or even multiple IT failure grounding a carrier the size of AA has got to be inexistent. I vote for the cover up theory.

Belgium Great Country ! Nobody within the thousands of AA employees, including the full reservations department and IT department, knew they had to have a plan B for eventual fail outs. Only the Belgians know this. Even more, we tell them how to do this:

If this is really an IT problem, it's a big cock-up by AA. Everybody knows that whenever an entire operation is supported by an IT system, you need to have a plan B. You need to have back-up systems that would allow minimal functionalities and provide information on bookings for the next 7 days.

Operating without a plan B is the perfect example of an amateur operation, regardless of any technical IT matters.

Belgium Great Country ! We offer the the solution: "just print it" !

Back in the days, each station would print early in the morning before the first flights, a list of all the departing passengers for the same day and the following day, so that in case of this kind of things, there would be a plan B. There would have been delays as each passenger would have to be checked in manually, but it's still much better than what happened today.

[Acid-drop]

I still beleive in what I know from the industry : people spend their life making sure such a thing is impossible. The only possible explaination is a voluntary shutdown, being affraid they would lose more than stoping completely the whole business. Data leakage, database corruption, national security reasons ? You'll never know.

[KriVa]

Is this system maintained by AA themselves, or is it a third party company doing this? That might also make a big difference.