This past week Tweetboard experienced a downtime period that extended over four days. Given the service level we have committed to provide, we believe that an explanation is owed to our loyal users.

For a long time we have been planning upgrading our database to utilize a table partitioning approach which would help better scale our storage hardware. We scheduled the upgrade for this past weekend.

The first stage went without an issue, the database was updated quickly and easily with less than 5min downtime while all services were restarted. On Saturday however, while working on the partitioning, the process required we drop the primary key index in our messages table and recreate a regular index instead. However, during the process, myisamchk ended up with a zero file size MYI for the messages table. This was a MAJOR PROBLEM as it meant that all of the messages (over 200 million posts) that we have built up over the past 7 months were gone. The same file was corrupted in our database backup during a simultaneous move due to space constraints.

A data recovery procedure would have required to physically remove the hardware to send it out, OR a remote attempt at recovering the data which would have cost $4K to $9K. Our best option was to attempt to recover the data ourselves.

We then moved to trying various “disk forensic” file recovering tools, including: “Foremost”, “TestDisk”, “PhotoRec”, “ext3undelete”, “sleuthkit” and others, but none were able to fully recover the data, due to the file being of such a big size, fragmented, and missing header/footer info.

We turned to manual analysis of the partition and even took the time to learn the ins and outs of low-level ext3 filesystem. The drive was instantly unmounted once we found about the corruption, which stopped data from being written over most blocks. That allowed us to reconstruct the chain of blocks, since we knew the location, size, and last deletion time of the file.

On Monday morning we had managed to recover 93% of the corrupt file which would in turn mean that approx. 90% of the actual database could be recovered. This was brilliant news but there was still a lot of work to be done and there was still a possibility that the database would not validate.

The team continued to press forward and yesterday (Thursday, 28th Jan) the recovery process had been completed and we could resume the original task of partitioning the database and updating the Tweetboard scripts to work with it. Finally, yesterday Tweetboard was brought back up and there was only one task left to do, write this blog and tell you our story.

What do we expect moving forward? Well, the database is back up as is Tweetboard, but we did still lose that 7-10% so we will be running background scripts over the next few days to recover all tweets that were lost.

We thank you all for your support and we do apologize for the downtime.

This weekend Tweetboard experienced some of the longest downtime in our history while we upgraded our storage system, giving us more room to grow.

When we started Tweetboard we never anticipated the rate in which our message database would grow, we showed you previously how quickly our message database was expanding, and trust me, it hasn’t stopped since then. In the two months since that post our database has continued to grow and is currently at 125million posts. It is great news to see Tweetboard’s growth rate, but we were running the service on a rather limited storage system, which though have great speed and reliability, was inadequate in size, really not leaving much room for expansion.

So now a brand new storage system has been installed giving us much more room to play with, however it wasn’t an easy upgrade process. We were unable to just migrate the data from the old storage system to the new one. The new hardware required a special controller, which meant we had to format the old storage system and reinstall everything fresh on the new system, including the OS, Web Server and all the core service components. So as you can see, a simple hardware update became a full blown system reformat.

Later this week we will have another *short* period where the service will be unavailable as it will be required to take one of the servers down to tweak areas of the new storage system. We appreciate your patience and thank-you all for your support!

We have heard your cries, and it is great to know that so many people can’t wait to get Tweetboard. So starting now we will be releasing a block of 2000 invites to users who have initiated their requests in the past few weeks.

The 2000 invites will be released slowly and you will be notified via Twitter once you are accepted. The approval “tweet” will come from one of two accounts, @microforum or @140ware, so if you’ve been anxiously waiting your invite, do keep an eye on your mentions stream.

Please keep in mind that due to Twitter’s limits on the number of posts we can send per hour/day, it may take a few hours/days to release the 2000 invites. If you still haven’t been accepted in two days time please open up a support ticket with a link to your invite request on twitter.

Going forward, we have seen that people are looking to get up and running with Tweetboard ASAP, so we have made the decision to automatically accept a small block of invites every week. We hope this will substantially reduce the time that you are waiting and keep our queue down.

Tweetboard has been growing for some time now, and it is about time to gain some personality, and one that you can design. What do I mean? Well, the Number one requested feature is about to be released to the public. You will be able to select from a list of official templates, or you can delve in and create a custom design that compliments your site layout.

Style your Tweetboard to match your site!

Before

After

For those that don’t want to dig into the CSS, we are including a few default styles. Activating these is as simple as selecting a pre-set style from a dropdown box and hitting save.

So this is where we all come in, we have created a section on our Wiki dedicated just to Tweetboard styles where you will find user submitted styles as well as tutorials on how to create your own style. We also want to see what you can create, so bring out your creative side and show us what can be done, once you are done, submit a new page along the lines of this one, with a description, screenshot and the CSS used.

The Tweetboard team has been working hard behind the scenes, and is getting ready to put a bunch of new features out the door.

While in the past we have released features to all TB instances simultaneously, we have determined that it works best when rolling out features to a smaller number of users. This allows us to a) test new functionality more extensively, b) tweak features on the fly with little/no impact to our general userbase, c) gauge our audience’s interest in a specific feature and d) overall, bring you a more polished product when each feature finally gets released.

For these reasons we have created the Tweetboard Pre-Release Team, a group of brave Twitter/Tweetboard users who will be getting immediate access to our roll-outs, once features have passed the initial QA phase. The group is currently composed of 44 users, however we are taking requests via our ticket system to officially join the Team.

Please feel free to submit a request to join the pre-release team if:

• You own a highly trafficked site (OR)
• You have 1000 (+) Twitter followers (OR)
• You consider yourself a “Conversational” Twitter user (you often reply to your followers, and they reply back) (OR)
• You consider yourself a “Broadcasting” Twitter user (you seldom reply to your followers, or you don’t get many replies) (OR)
• You have previous QA testing experience (and are good at it!) (OR)
• You don’t meet any of the above requirements, but you are a BIG Tweetboard fan

Click Here to Join the Tweetboard Pre-Release Team

Note: Approval is not guaranteed. Please be patient, we are processing these requests manually.

Our current pre-release team members are:

mobile_x
TrinityMMo
walidmrealtor
seifertj
RNchat
CuteGecko
bluetoquetips
matthewscoaches
mtm_chicago
asltweets
fcaps
dramativity
ChristianBoys
xcheque
SiteSuites1
parents_help
Aarinfantasy
digibillydesign
derigel
KingPinX
doctorkaraca
vBHispano
AndrewAWilson
billwesterman
ChristnImpctNws
humanbeatboxcom
CastleType
4sixisix
deuxdoom
FlameHabbo
PhilCoyne
DevDroidian
StayBankdotcom
elm6r
WeRLife
LouisC
RolePlayGateway
CezzBB
Retropinuplady
connectgaming
aznextgov
Superchmed
PhilBaumann
obtainer

If you follow Twitter related news, you may have read this article on “API name squating”.

As you probably know, Tweetboard has been honored with being a finalist at the Launchpad contest at the Twitter Conference in LA next week.

Well, we were going to hold this announcement until the conference, however, since we’ve been advised that this is becoming a common practice among Cyber/Name Squatters, I’ll give you a solid hint right now on what’s coming!

And here it goes: I strongly suggest that, if you use Tweetboard, or plan to do so in the near future, you register your own Twitter application YESTERDAY. Furthermore, I suggest that you register 2 apps: one named as your website (MySite.com) and one named as your Brand (i.e TacoBell).

Register your Twitter Application now

Why? Well, I’ll let that part to your imagination, OR better yet, just don’t think about it, go ahead and register your application right now! On Sept. 22 (next Tuesday) you will know why!

Yesterday Oleg pulled a graphical representation on the growth of Tweetboard’s dB. It sure is interesting… it appears our dB is growing at near exponential rate.

Twitter’s API has proven exceptional at serving a LOT of data at amazing rates, especially after we implemented the Realtime Updates notification system. Thank-you Twitter.

I’m really happy with the growth. On the same token, I realize (or maybe I don’t! ) the challenges that this represents moving forward.

Wish us luck!!

We are testing a new functionality (to be announced soon), where you will be able to reply to anyone in the Twittersphere, and the original post, along with all replies, including yours, will get “pulled” to your Tweetboard, with the original post showing up as the parent Tweet… pretty awesome.

You will truly be able to “Bring the Twittersphere to your Content” with this feature.

To this effect, I have started following a few “celebrities” via our testing account, and I must say I’m trully amazed at the number of replies some of them get!

Check out how many responses PDiddy (@iamdiddy) got to his tweet on the incident that occurred during the Session of Congress, earlier tonight… Whoa!! 177 tweets and counting.

How many do you think President Obama (@BarackObama) got for this tweet?

We are happy to announce that Tweetboard.com now sports a full featured bulletin board located at bb.tweetboard.com.

Having worked with forum software development for over 5 years, we know the power of community, and more specifically, the benefits that building a community around a web product/service can bring, to both, developers and users alike.

I’ve created a short intro post to our new BB (a.k.a forum) here:
http://bb.tweetboard.com/topic/welcome-to-the-tweetboard-bb-aka-forum-1

Feel free to stop by, and join us in our new community!

PS. The board (powered by awesome bbpress) is integrated with our wordpress install here, therefore you can register here (or there), and your profile will be shared, including your avatar, etc. Works with gravatar too!

Tweetboard Alpha 2 will be debuting in the upcoming hour to over 10 thousand sites near you.

As stated in our previous blog post, Tweetboard Alpha 2 is a full rewrite to the underlying code, resulting in a faster, leaner and more secure Tweetboard. However, the two most notable updates in Alpha 2 are a REDESIGNED Notifier Tab and our REAL-TIME Update Notification System!

The original tab design, although very clear and bold, was not ideal for all types of sites. We realized this shortly after deployment and immediately started working on a revised tab design that would appeal more universally, while being less intrusive. We went through many iterations during the process, but it definitely paid off as we are extremely happy with the end result.

Tweetboard Alpha 2 also features an awesome new REAL-TIME notification system. The original notification system required a page refresh to display “new tweet(s)”, which took away from the “fast-talk” potential of your Tweetboard experience. The new, updated notification system will show you what’s new, *the second* it occurs, both, for updates posted via Tweetboard, AS WELL AS updates posted via Twitter. We feel this is truly a game-changing update, and once you test it out, I’m sure you’ll agree with us.

The most notable updates:

1. Tweetboard Alpha 2 adds COMET technology (more info) to push updates in real-time. “New Post(s)” notifications will no longer require page refreshes. This update adds a new dimension to the TB experience, making it “sort of” addictive, seriously!!
2. 140Ware has received upgraded Twitter API access that will allow *instant* pulls from your timeline. Delays in pulling Twitter timeline updates has been the #1 complaint in Alpha 1, we went from ~1 minute updates to ~2-3 hour updates after the initial launch period due to API limits.
3. Replies to replies and replies to those replies, and the replies to those replies (you get the idea) will be pulled now. Previously it would pull first level replies only, and unlimited nesting for posts published via the Tweetboard. This one is key.
4. To address user’s #2 complaint, we have created a refined notifier tab, especially designed for sites where the original (bold) tab is not ideal.
5. Alpha 2 introduces significant usability updates and an updated default sorting option to encourage activity (latest replies bump “threads” to the top of the list)

We hope you enjoy the new and improved Tweetboard!