Missing E-mails Unearthed...

Status
Not open for further replies.
Joined
Feb 6, 2010
Messages
4,836
Location
Central Texas
http://www.washingtonexaminer.com/30000-...556522/comments

Quote:
Up to 30,000 missing emails sent by former Internal Revenue Service official Lois Lerner have been recovered by the IRS inspector general, five months after they were deemed lost forever.

The U.S. Treasury Inspector General for Tax Administration (TIGTA) informed congressional staffers from several committees on Friday that the emails were found among hundreds of “disaster recovery tapes” that were used to back up the IRS email system.
.
.
.
In all, investigators from the inspector general’s office combed through 744 disaster recovery tapes. They are not finished looking.

There are 250 million emails ion (sic) the tapes that will be reviewed. Officials said it is likely they will find missing emails from other IRS officials who worked under Lerner and who said they suffered computer crashes.

I searched for the previous thread on this but searches for IRS returned no results.

I'm interested what those of you who know far more about this than I Re: server backups of any organizations email records have to say. I do find it odd that the first was all HD's were destroyed. I know better that servers take care of this task, not indiv. user's HDs. Second, that tape is still used. Third, did it really take this long? Tape is sequential though, so it might have.

While this can quickly get political given well... the current politik. I'm more interested in the technical side...of how any organizations emails, records, data flow, etc. are managed/archived/recorded in a digital format.

So IT guys/gals...chime in here and illuminate us.
 
The vast majority of voters have probably have forgotten Ebola! Don't wake them up with the IRS Deleted emails
sleep.gif
 
Originally Posted By: CT8
The vast majority of voters have probably have forgotten Ebola! Don't wake them up with the IRS Deleted emails
sleep.gif



They should just have an illegal Mexican who just got amnesty infect Lois Lerner with Ebola upsetting LeBron James so much he was unable to score more than 15 points in a game causing the stock market to go up thereby allowing Banks to loan again to sub-prime borrowers and the GOP to win landslide elections and take control of the Senate.

Then I could keep up.......
12.gif
 
As I stated above, this could quickly get political. That's why I posted it in the confuser forum. I'm wondering from those in the know. Enough of politik, they'd argue with a right triangle or a plumb bob. I'm an engineer so lets deal with facts.

How fast could an (comptetent) IT type full fill a search like this? Would an index be available? Or a hash tag? Yea, nay or no way? What say you?
 
Originally Posted By: CT8
The vast majority of voters have probably have forgotten Ebola! Don't wake them up with the IRS Deleted emails
sleep.gif


Too late....rise and shine!
 
Well, importance of backups flow from the top-down. (Yes, tapes are still used, and there are positives to them. You aren't really a server administrator until you've done the math and tuned a system to keep a tape streaming.)

It's a thankless job. I've heard it compared to the guy who
carries the launch keys in a nuclear sub. You can only eff up, you'll never get a promotion out of it, you don't get good shore leave because you always have to be within a few minutes of the boat in case of a need of a launch. Your mind focuses on how to pass the keys off to the next new recruit, so you can make your life better.


As far as the story, it's (another) Friday night news dump, shortly before Thanksgiving. Trying to bury it, among Cosby and Ferguson grand jury.
 
Yo Doog...let's stay on topic, OK? No drive-bys. Make a hole for those in the know! The topic has been dilluted enough as it is, thus the request for facts.
 
You have to have policy for this sort of thing as there's confidential data at stake. For every lost email there's "found" 2 million social security numbers.

So while they should be off site protected against computers crashing, they should also be protected against burglers.
 
Not getting political, but the whole "our hdd were destroyed" excuse was terrible.. they should have seen major jail time.

Maybe "they" figure it will blow over now that elections are over?
Seems really fishy about the timing but who knows.

I'm guessing disaster recovery tapes have excellent physical security.
 
LOL! So very true and so very sad.

Originally Posted By: CT8
The vast majority of voters have probably have forgotten Ebola! Don't wake them up with the IRS Deleted emails
sleep.gif
 
Who has time at work to write 30,000 E-mails!?!?!

Aren't they suppose to be "working"??
 
I was going to skip paying my taxes on April 15 and give the IRS a "my hard drive failed" type of excuse. I guess now they can tell me that everything on a failed hard drive can be retrieved and I'll have to try something else.
 
Originally Posted By: sleddriver
As I stated above, this could quickly get political. That's why I posted it in the confuser forum. I'm wondering from those in the know. Enough of politik, they'd argue with a right triangle or a plumb bob. I'm an engineer so lets deal with facts.

How fast could an (comptetent) IT type full fill a search like this? Would an index be available? Or a hash tag? Yea, nay or no way? What say you?


I have no idea on how long to restore this data as I am not a server guy.

However I did work on an archival product for email(compliance) reasons with search for Lotus Notes and Exchange(either likely at IRS) and it would take a while to backfill it. If possible with security I would actually pick the Google Gmail product as archive for obvious reasons in search.
 
The short answer is "way too long"

The amount of time it took for this seems extremely long. Something more along the lines of a couple weeks would be acceptable, even given off-site tape backups. We're talking 6+ months now, the government should fire their data warehouse.

Everyone knows the emails for an enterprise mail system are all server side. Of that there is no question. Hard drive destruction was about the most obvious cover story possible.

They had a date range and they had a username. This amount of time to recover the emails from backup is unacceptable by any metric.
 
I have the rare perspective of working on both servers and tape as our customer service functions were combined. So on any given day, I might work on an Oracle/Storagtek SL8500 tape library complex and then an Oracle/Sun M6-32 server.

The SL8500 can be configured into a 10 library string.

http://www.oracle.com/us/products/servers-storage/storage/tape-storage/034341.pdf

There could be over 100k tapes in a 10 library complex. That doesn't even count tapes the customer has moved to off site storage.

It could take a while to find someone's mailbox in such an environment.

Am I saying this is what the IRS has? No. I'm simply saying it may take a while to recover data if you don't know which tape or tapes have your data.

But like Subdued says above, this should be a matter of days and at the most weeks. Not months!

Not sure I'm buying the hard drive destruction/failure aspect either. Wouldn't such a critical server have some sort of RAID protection in addition to backups?
 
Originally Posted By: javacontour
I have the rare perspective of working on both servers and tape as our customer service functions were combined. So on any given day, I might work on an Oracle/Storagtek SL8500 tape library complex and then an Oracle/Sun M6-32 server.

The SL8500 can be configured into a 10 library string.

http://www.oracle.com/us/products/servers-storage/storage/tape-storage/034341.pdf

There could be over 100k tapes in a 10 library complex. That doesn't even count tapes the customer has moved to off site storage.

It could take a while to find someone's mailbox in such an environment.

Am I saying this is what the IRS has? No. I'm simply saying it may take a while to recover data if you don't know which tape or tapes have your data.

But like Subdued says above, this should be a matter of days and at the most weeks. Not months!

Not sure I'm buying the hard drive destruction/failure aspect either. Wouldn't such a critical server have some sort of RAID protection in addition to backups?


My HP tape backup has a lot of tapes too - but all I need to do is run a command and point to the file or location I'm interested in. Then I get a list of tapes, along with the dates the versions of the data on them came from.

All of our tapes have identifiers and are stored in cabinets based on that. So I go to the room with the right cabinet, open the drawers until I find the range of tapes I want (they should be labeled on the outside, but they're not. But our cabinets are four drawers high so it's not too big of a hassle). Look down the row of tapes, find mine, then I feed it to the robot and get my data.

Is the Oracle system more cumbersome?
 
Last edited:
I don't think so. It's all a matter of making sure you don't lose the media. That you have a good means of finding media not in the library complex, etc. I know the hardware side of tape much better than the software. Occasionally, I have to help a customer with the software side of things. But the bulk of my tape calls are down drives, robots or stuck tape. Replaced a CAP today on an L700 and tested a couple drives for another. Tapes go in. Tapes go off site. Hopefully each customer has a way to track which volumes are off site at vault providers such as Iron Mountain.
 
Not providing excuses, but anyone who works in a big IT shop knows that a great proportion of stuff implemented isn't done they way it should have been done, that original considerations of stuff we "want" quickly get discarded as schedules slip, budgets increase.

Rely on others "to do their job" and the possibility for things not working well down the line rises.

Often backup systems are upgraded with little regard to maintaining access to the data stored on previous backup systems archive tapes. I just EOLed a CAD system where I work and there were 8mm, 4mm, DLT and QIC250 tapes all holding archive data. Weeeee. Archival backup is marginalized @ all levels of mgmt. Ya, there are problems getting stuff off a 1994 tape unless someone keeps a pile of old hardware and OS disks (which i did) So the data archiving house might have supplied the tapes, the ability to read them, well.....

Quote:

This amount of time to recover the emails from backup is unacceptable by any metric.


Not really; recovery times can be dictated by SLA. Climbing back months for email trails is usually a result of a legal discovery which could be part of a multi year lawsuit and might be very infrequent. I don't want the gov't paying for two week turnaround for mountains of archive data they will never access.

However, if they are part of DR tapes, obviously i agree with your assessment.

http://www.computerworld.com/article/254...ead-never-.html

Quote:

Largely due to its cost and transportability, tape remains the most common choice of media for long-term data retention. However, this may be mainly an artifact of the still prevalent practice of "backup as archive." In this situation, tape essentially serves as "WORN" (write once, read never) storage, since once it's sent away to an off-site vault, the real hope is that it will never need to be recalled again. If this weren't the case, then organizations would have comprehensive programs in place to regularly recall and refresh these archival tapes, and we know that this rarely happens.


The only archive data I manage is stored on WORM NAS storage; the admins and the NAS admin can't even destroy the filesystem; this software was designed for Sarbanes Oxley compliance; The retention times for my data is > 50 years and data is real time replicated to a remote site; The data has been 'moved forward' several times as technology has changed.

All my other data is on dedup disk based backup with 30 day retention.
 
Last edited:
Update:

http://www.washingtontimes.com/news/2015...e=all#pagebreak

Quote:
Investigators have already scoured 744 backup tapes and gleaned 32,774 unique emails, but just two weeks ago they found an additional 424 tapes that could contain even more Lerner emails, Deputy Inspector General Timothy P. Camus told the House Oversight Committee in a rare late-night hearing meant to look into the status of the investigation.

Quote:
He said they have also discovered the hard drives from the IRS’s email servers, but said because the drives are out of synch it’s not clear whether they will be able to recover anything from them.


Quote:
The IRS belatedly told Congress it may have lost some of Ms. Lerner’s emails after her computer crashed, and asserted that the backup tapes didn’t exist.

But under questioning from Mr. Chaffetz, Mr. Camus said it took him only two weeks to track down the backup tapes, and when he asked the IRS depository for them, the workers there said they’d never been contacted by the agency itself.


Quote:
Mr. Camus said they were clued in to the 424 new tapes they just found a couple of weeks ago after realizing the IRS hadn’t given over a key document. They demanded that document, and realized it showed hundreds of other tapes existed.


Very interesting. So now the 'missing files' are being unearthed. Further, now we're hearing about 'tapes' and hard drives from email servers instead of from just her laptop. As we knew all along. Also...the workers at the IRS depository said they'd never been contacted by the agency itself! Sounds to me like Kosein/kosign just pergured himself. I wonder when he'll be called back.

As we knew all along, there must have been backups somewhere. And they're still being found! The IRS has also been clearly withholding evidence.
 
Worst case...Lerner will spend some time in jail. The present administration will be able to survive the next 2 years. Nothing will have changed. It never does.
 
Last edited:
Status
Not open for further replies.
Back
Top