Even More MySQL Moves!

We will be moving more customer database servers Friday night, November 7, starting at 10PM PDT. We expect the move to take 3 hours, ending at 1AM PDT on Saturday. Each server will likely be down a fraction of that time.

The following servers are affected:

snowball garfield meow toonces luna guinness hobbes orion dandilo cassandra figaro mimsey rumple burbank aslan roary natty hooch cujo bingo sekhmet sanspoof sparky barbaro seabiscuit cykotix masevo fwap naroga petrelli bennett grendel lingling puff ein

We appreciate your continued patience as we work to streamline and reorganize our server layouts. Also, I apologize for the late notice of this maintenance window, I apparently didn’t notice that WordPress had eaten my previous post. If you have any questions regarding any of this please contact support.

Happy DreamHost Movin’ Team!

Update — Sat Nov 7 00:56:36 PST 2009 — All clear, the move is over and every MySQL server is responding normally.

Comments: 31

Posted In: General Outages, Multiple Server Issue, Mysql Outages

Posted By: micah

Jasper is offline

The machine is not responding to the remote reboots that are being issued. A member of our administration team is currently investigating this matter. The machine should be back online shortly. Unfortunately, I cannot provide an exact ETA. We truly apologize for the inconvenience this has caused. Please check back for future updates.

UPDATE 2009-11-06 12:24PM PST: Jasper is back online. It just needed a manual reboot. We will continue to monitor the stability of the machine. Again, we sincerely apologize for the inconvenience this has caused and appreciate your continued patience. If you are still having problems, please submit a support request.

Comments: 0

Posted In: General Outages

Posted By: brians

PS host instability issues.

We have a couple of hosts that are running into kernel issues which is causing downtime for Private Server guests on those hosts. We’re working with the VPS development team to generate a solution that does not undermine security on those machines. We’ll update this post as soon as we have more information.

Update (2009/11/06):

We’ve got a patch that should let us isolate and correct the problem we’re having – it’s not as simple as rolling back the kernel change due to a vulnerability with the previous version (it wouldn’t be secure to just revert) so we have to get this fixed. We’re doing our best to get it corrected as quickly as possible.

Comments: 30

Posted In: General Outages

Posted By: mir

Pistone server down

We are currently investigating an issue with the Pistone server, which is causing downtime.  We apologize for the inconvenience and hope to have the issue resolved shortly.  Please check here for updates.

UPDATE 1:20pm PST – We are getting ready to move Pistone’s data to new hardware. Sorry about the continued inconvenience, we’ll update this post again with any progress made.

UPDATE 4:10pm PST – Unfortunately, the first replacement machine we tried didn’t work out. We’re getting another one ready for Pistone, and will attempt the restore from backups as soon as the new hardware is ready.

UPDATE 6:25pm PST – The new machine is up, and all apache web services are running. We’re looking into the current high load, so this post will not be resolved until we made sure the server is running smoothly. Thanks for your patience!

Comments: 40

Posted In: General Outages

Posted By: patrickm

Homie Mail file server move 2009-11-05 10pm pacific

As part of our rack consolidation efforts we will be moving two four of the file servers which provide storage for homie mail on Wednesday 2009-11-04 2009-11-05 at 10pm pacific. This will result in some downtime for customers on those file servers. The file servers we’re moving are milesaway, fromordinary, tastesgreat, and lessfilling. The estimated downtime for this move is approximately 2 3 hours, during which time your incoming mail will simply queue up on our mail servers waiting to be delivered.

We thank you for your patience and apologize in advance for any inconvenience this downtime may cause.

Update — Wed Nov 4 18:00:53 PST 2009, we have postponed this move until tomorrow night. The post date reflects the new maintenance window 2009-11-05 22:00:00 – 2009-11-06 00:00:00. We apologize for the short notice change in plans.

Update — Thu Nov 5 22:02:58 PST 2009, we are beginning the moves now. We will post an update here once everything is finished! Thank you for your patience and we apologize for any inconvenience.

Update — Fri Nov 6 01:20:48 PST 2009 Everything is back up and running! If you are still having any issues, please contact support!

Update — Fri Nov 6 02:06 PST 2009 The moved mail servers are back up, but an issue with our authentication servers was just identified.  We’re working on a resolution now and will update this post when everything is working again.  Thank you for your patience during this time.

Update — Fri Nov 6 07:11 PST 2009 – The lingering authentication issues are now resolved.  We truly apologize for the inconvenience.  If you are still having issues with any of your hosted email accounts, please contact Support and they will be very happy to assist.

Update — Fri Nov 6 8:00 PST 2009 – We sincerely apologize, as there is still an issue with one of the mail servers in the Homie cluster that is causing email/webmail downtime.  We are resolving this issue as I type this, so we expect it to be resolved shortly.  Again, our deepest apologies about this continuing issue.

Update — Fri Nov 6 11:20 PST 2009 – We believe we have identified the problem, and are working on the resolution. We’ll update this post again with any progress made.

Update — Fri Nov 6 1pm PST 2009 – The problem causing authentication, sending and receiving issues on your mail servers are now resolved. We’ve been testing numerous accounts in the past hour to make sure, and everything appears to be working again. You may experience short delays – connections may clog up a bit initially due to the rush of sending and receiving after the services were restored. If you’re waiting for mail that was sent to you during the data migration, those are being queued either on the sender’s server or on our incoming mail servers, and should be delivered shortly – unless they bounced back to the sender immediately after the first try. Sorry again about the inconvenience this mail downtime has caused. If you’re still having problems with your mail, please submit a support ticket with details, so we can look into it for you.

Comments: 123

Posted In: Email Issue, General Outages, Multiple Server Issue, Webmail Outages

Posted By: kitchen

Yankees cannot buy uptime!

We apologize for the issues customers on the ‘yakees’ server have been experiencing. The problem was due to hardware (the RAID card to be specific) but we have that corrected and are bringing it back up now (it’s just taking a bit longer than normal as we’ve had to go through the post deployment process). It should be available in the next 20-30 minutes but we’ll update this post once we have it back up again.

Update (15:45 PST):

Yankees is back up and running and should be fine but the admin team is keeping an eye on it just in case.

Update (2009/11/06 8:15 PST):

Yankees is back up again and we’re keeping an eye on it to make sure it remains stable. Basically the raid card was having issues with one of the defined devices which meant we needed to power cycle it but it looks all better.

Update (2009/11/06 10:05 PST):

The issue is recurring so the admin team is looking into whether or not we need to move the machine to new hardware – will update this post once we have determined the best course of action.

Update (2009/11/06 13:00 PST):

It is up currently but it refuses to remain stable so we’re in the process of replacing the hardware entirely (the admin team just needs to put together a replacement machine as we didn’t have one ready to go like we usually do). They’re going to get that taken care of as quickly as possible and we’ll update the post here as soon as it happens

Update (2009/11/06 16:10 PST):

Yankees is back up on new hardware and will be stable – for now we have mounted the data from our backup server and are copying it over to the local disk which may effect performance slightly until it completes but the server should be back up for good. We apologize again for the inconvenience due to the hardware failure.

Comments: 4

Posted In: Single Server Issue

Posted By: mir

Bulger is unhappy

We are in the process of moving everyone on bulger to new hardware. The filesystem kept mounting read-only, which continually was making things unstable for the customers using the machine.

We’ll try to make this as quick and painless as possible, but it will take some time. Please watch this space for updates.

This only affects the web server. Customers on bulger will have access to their email and databases.

Update 5:20 P.M. PST: We are still restoring data on bulger. Once data is restored from backup we will do a final sync with the data from the old machine.

Comments: 20

Posted In: General Outages, Single Server Issue

Posted By: Terri

Multiple Server Reorganizing

Tonight we’ll be moving 26 of our web servers from one rack to another. The destination rack is fully prepped so downtime shouldn’t be more than 10-20 minutes per server. We will begin at 10pm PST and will update this post with the all clear.

Sorry for the last minute post, we’re having to shuffle quite a few things we hadn’t planned on.

Here’s the list of affected machines:

bitters krabappel clancy arnie ned dawber chalmers bart shemp salt manjula abe brockman gimli mona patty jebediah brawndo jimbo quimby duffman drederick jasper mccallister

Update 11:20pm pacific: And these are all done! If you are still experiencing any issues, please contact support!

Comments: 6

Posted In: General Outages, Multiple Server Issue

Posted By: micah

fscking bulger

The machine ‘bulger’ is having some problems with its filesystem and we’ve taken it down to run an fsck on it. We will post updates here as things progress.

Update Nov 5th, 11:30am (PST): The file system check on this server has completed. We are now in the process of moving this server to new hardware. Please keep updated at this new post:

http://www.dreamhoststatus.com/2009/11/05/bulger-is-unhappy/

Comments: 28

Posted In: General Outages

Posted By: kitchen

Homie downtime across various machines

We had to upgrade the kernel on several of the homie machines. A reboot is required upon completion. However, a few machines are still having problems. We are looking into those machines now. The machine whose kernel did not seamlessly upgrade will be reverted back to the previous kernel version. Here is a list of machines that we have attempted to upgrade. We hope to have service restored ASAP. Sorry for the inconvenience this has caused. Your patience is truly appreciated. Please check back for future updates.

addisababa alouettes andross angels argonauts asmara avengers baku bandar bayhawks bears beijing belmopan bengals biggs bills blackhawks blazers bluebombers bluejays bobcats bogota brasilia brazzaville brewers bridgetown broncos browns bruins brumbies brussels buccaneers buchalter bucks buenosaires bujumbura bulls burke cairo canberra cannons cardinals castiglia cavs chivas clippers cohen colossus colts coonan copenhagen cowboys crew crusaders cyclops d-fenders dedede dhaka diamondbacks dili dio djibouti dodgers dynamo eagles earthquakes ecco escobar eskimos expos falco falcons fever fire fitzpatrick flames force gaborone galactus galaxy gambino genovese ghostrider giancana gotti gravano greene greengoblin grizzlies havana havok hawkeye hawkman hawks heat herediano highlanders hobgoblin hoffa hornets hulk hurricanes iceman indians islanders jaguars jazz jets joker jubilee kabul kakuji karpis kings kinshasa knicks lakers lansky lapaz legs lepke liberty lightning lions lizards luanda luthor magic magneto malabo manama mangano mariners marlins masseria mcgurn mclean mets monarchs moran moroni mrfantastic nassau nationals ndjamena nets nicosia nightwing niners nitti nuggets obelix oilers olympiacos ottawa ouagadougou outlaws pacers packers padres panthers patriots penguin penguins peppy phillies phnompenh pirates pistons porto-novo prague praia preacher predators pride provenzano quicksilver quito rams rangers rapids raptors ravens rays reds redwings revolution robin rockets rockies roseau rothstein roughriders royals sanjose sansalvador santiago santodomingo saprissa sarajevo schultz seahawks shadow shock silversurfer slippy snowbird sofia sox sparks spiderman spillane spilotro spurs stampeders stjohns sugeknight suns suva tallinn terranova texans thimphu thing thrashers thunder tick tigercats tigers tirane titans twins venom vikings walsh waratahs wario warriors wasp watanabe wizards yamoussoukro yankees yaounde yerevan zagreb zwerbach

UPDATE 12:31AM Pacific: This actually also applies to a number of servers under other clusters.  Some of the servers under Blingy, Postal, Randy, Frisky and Spacey were also updated with the new kernel, which will have caused some downtime as we rebooted the servers.  We still have a small number of servers that need this update, but we are done with the updates for the night.

If you are having any issues with your hosting, please don’t hesitate to contact Support for assistance.  A new post will be added for the updates that happen tomorrow, so please check back for that.

Comments: 16

Posted In: General Outages

Posted By: brians