moblog uk

moblog news

by moblogmods

user profile | dashboard | imagewall

« older newer »

add moblog news to your friends

Moblog is about sharing, communicating, playing with new gadgets, and above all having fun!
We love this site so much, we decided to give up some time everyday to help manage it.

Welcome to moblog news. Your community-powered site support hub.

This is one of three moderator-only moblogs, set up by the moderators for all mobloggers.

We are going to use this blog to publish all appointments, decisions and policy changes made between the moderation and site admin teams behind the scenes.

Follow this group to get updates on how the site is developing and to get an insight into what goes on to keeping the good-ship Moblog sailing!


Meet the people powering moblog at meet the mods

Meet other mobloggers in Interview52

Hooked up with any mobloggers recently? Post it in moblog meets.

Are your pictures good enough to win a competition? photography competitions.



If there is an event, appeal, point of contention or call to action which you feel is strongly relevant to the Moblog community and you would like it posted here please contact Viv or nige or any of the moderators, to broadcast your message via moblog news.



flash widget

Search this moblog


Recent visitors

Sluggish speeds and crashing...

(viewed 3556 times)
Bookmark and Share
As there have been a number of occasions over the past few months where users have experienced slow page loading and timeouts - without too much information coming back, aside from "We're looking into it..." we've spoken with Mat, to try to get a better understanding of the problem.

Hopefully, Mat's explanation below will give you an insight into the current issues and also help when we ask you to "Bear with us. We're working on it!!" :)

Mat said:

"The problem appears to relate to the database server. Sometimes a query will just not finish, and that lock-up gradually leads to other processes building up and a slowing down of the server to the point where it feels sluggish, or just won't respond at all. For some reason this query doesn't time out as it should do, nor does it complete.

We've never fully determined what makes this problem happen, and believe me, both me and Dan have put huge amounts of time into that job - logging stuff, reading documentation and logfiles, posting to various forums and newsgroups (mostly MySQL and Django ones). No joy. It's not like either of us are inexperienced developers either.
It's the main reason the new site is so unpolished - an awful lot of money went on paying Dan to try and figure out why the site kept "crashing", money and time which really needed to be spent on refining the site's features. Then we ran out of cash before getting the issue properly sorted.

We made a number of changes to the site and its hosting setup, some minor, some more major - all appeared to reduce the problem, but not remove it entirely. In the end, Dan created some scripts that keep an eye on how things are running, and auto-restarts stuff if memory use gets out of control. This is far from an ideal solution and to have to stoop to this level upsets me a great deal. I'd always rather fix a problem at source than hack my way around it.

Most of the time the problem resolves itself - again, we're not entirely sure why this happens, but it does - or is resolved after one of our auto-monitoring scripts kicks in.

Any suggestions from MySQL/Django/Linux gurus out there would be much appreciated."

Posted by 540air

Thanks to parabolichobo for the very apt image.

NB If you haven't already done so - please add 'moblog news' to your friends list.
2nd Sep 2009, 14:31   | tags:,

Factotum says:

Thanks, you guys!

2nd Sep 2009, 14:44

Twiglet says:

Euwwwww! Slugs are yuk!

2nd Sep 2009, 16:59

nige says:

I quite enjoy them in a curry.

2nd Sep 2009, 17:05

Twiglet says:

Oh groooooooosssss

2nd Sep 2009, 17:18

JokerXL says:

Have you tried switching it off and then switching it on again?

. . . . I'll get me coat.

2nd Sep 2009, 17:42

JokerXL says:

ps.

did you know a "slug" in Dutch is a "naked snail"

which is logical, I thought.

2nd Sep 2009, 17:52

jc1000000 says:

ROFLMAO @ Joker "switching it off and on again"... if only you knew... if. only. you. knew.

:D

2nd Sep 2009, 18:53

Twiglet says:

Dunno what you're laughing at jc. That's what works for the rest of us.... : )

2nd Sep 2009, 18:59

rhys says:

I'll just have to pretend I know what even this simplified version means :)

Keep up the good work!

2nd Sep 2009, 19:00

Nice to be kept up to date, thanks.

As for Jokers amusing remarks isn't that what they said they have had to resort to as a fix ????

And the non technical guy came up with it straight away ;)

2nd Sep 2009, 19:04

Caine says:

Everything is functioning and moving at a good pace today. It's soooo nice when that happens.

2nd Sep 2009, 19:13

I like to give the kids a bag of slugs each and tell 'em they're Pontefract cakes!

2nd Sep 2009, 19:58

Now we're on this subject I can be more specific. Whenever I click on my dashboard link, the speed of loading does vary slightly but it's ALWAYS very slow. Navigating from there to a friend's blog is perhaps slower than it should be, but nowhere near as noticeable as my dashboard, and this is the case every time I go back to it. The latest and Home links are also slow, but again, not as horrendously slow as my dashboard. I don't know if this helps at all, hope it's of some use.

2nd Sep 2009, 20:02

nige says:

Same for me as well, 'hobo. From what mat said, my understanding is that anytime a person clicks their dashboard it is a much more complex query than simply clicking latest (because it has to go through your entire friends list to see who has updated, and then present that information to you). Sounds like what mat has explained x 10 basically.

2nd Sep 2009, 20:48

jc1000000 says:

Yeah, dashboards is where we have the most problems right now. For instance, you should be returned to the page you were on when you log in but currently it sticks on the dashboard.

Busyness is the most complicated & server intense query but i think the dashboard is probably not far behind.

2nd Sep 2009, 22:21

Puddlepuff says:

Isn't there a way of simplifying the query?

3rd Sep 2009, 12:10

mat says:

PPuff - two things there:

1. it's not consistent. if it were a single query, or class of query, it would be relatively easy to fix. it doesn't appear to be a particular one though.

2. Django's object relational mapper, which abstracts database queries from the programmer (ie, you don't write raw SQL, you just say "get me stuff that matches these criteria from the database") makes this kind of hand-optimisation rather difficult. I used to tune every individual query on moblog 1 at an SQL level - ie, running the query straight on the database, benchmarking it, tweaking it until it's as fast as possible - but that's less easy to do in a modern web framework system like django.

There's a bit more (rather technical) information on Django's ORM and some of it's problems here

Personally, I prefer to design my own schema, and queries, by hand. But then the leverage django provides for rapid/reliable development in other areas can balance out it's (relative to hand-tuned) lack of refinement at the database level. Of course, throwing more raw computing power at the problem would help too. If we could afford a beefier server and/or cluster of servers, I'd do that.

4th Sep 2009, 14:21

mat says:

Nige, hobo:

"anytime a person clicks their dashboard it is a much more complex query than simply clicking latest (because it has to go through your entire friends list to see who has updated"

In theory*, it should be as quick to draw your friends latest posts than to draw the latest page. There's a quick select to get a list of friend's ids, but that's probably covered during the getting of user's info anyway; then a select based on those ids to get a list of posts. After that, it's exactly the same process as latest or home pages, but with fewer other things on the page. But that's all easy stuff (searching post ids is fast - 'cos they're integers and integers index well). Searching text is where things get SLOW.

I suspect dashboards get slow checking for updates on latest comments. And possibly get slower the more friends you have. Maybe even the number of friends you have who use privacy settings would affect it too. My dashboard isn't that slow, it's slightly slower than other pages, but not much.


* ie, "if I'd made it", am not entirely sure how django abstracts certain things. It should be reasonably efficient - after all, it was designed to make website with - but you never know.

4th Sep 2009, 14:27

Puddlepuff says:

"anytime a person clicks their dashboard it is a much more complex query than simply clicking latest (because it has to go through your entire friends list to see who has updated"

Isn't dashboard the latest page where user = friend?

4th Sep 2009, 14:58

mat says:

Yes. Broadly speaking. Here's some pseudocode for anyone who speaks a bit of SQL (assuming here that $usersid is the userid of the person who's dashboard it is)

"SELECT $friendids FROM user_data WHERE userid = $usersid"

then

"SELECT $posts FROM posts_table WHERE post_owner_id IN ($friendids)"

and finally

"SELECT $postdata FROM posts_table, user_data WHERE post_id IN ($posts)"

$postdata is stuff like post date, title, imagefile name, etc. In real life, obviously it's a bit more complicated than just the above, but you get the idea.

Roughly speaking, latest is indeed the same as dashboard, but without the first query. But then you have to consider that there are other parts of the page too - dashboard doesn't have news, highlights, random (random is a Very Hard Query, it's actually generated once a day with an offline script). But then latest doesn't have recent comments with "updated" stuff - and that's one query per-comment, rather than the "recent comments" list on latest, which is one query for the whole list.

Anyway. As I said, it's not one particular query or even class of queries. It's just WEIRD. So many components have been replaced or changed, you'd have thought we'd have pinned down the issue by now. But apparently not. We even changed the whole webserver - used to be apache, now it's all done with lighttpd (apache still sits out the front and reverse proxies, but lighty+fastcgi does all the actual work). and the database went from myisam to innodb (that helped a lot, but didn't stop it). Then we replaced the django-to-database connector. And a bunch of other stuff. The list goes on.

Still, any suggestions or ideas will still be gratefully received, and I'll try to answer any questions you guys have too.

4th Sep 2009, 17:00

Add a comment


(P) what's this?

Track updates to this post with rssthis rss feed