How to write imageboard

Hello,
I found out that kusaba x is pretty old shit. What's this whole think about generating html files? Can't it be done simply by sql queries every time and do it with some ajax? How would you write it?

http://wakaba.c3.cx/s/web/wakaba_kareha
All you'll ever need.

How would you write an imageboard?

Very carefully.

There's a very good reason why the high-traffic boards all generate html at post time. The content is going to be exactly the same every single time the page is fetched, and reads happen much more often than writes. The way imageboards are structured makes it dead simple to offload the page-serving to a fully static server on a different subdomain than where the form submissions are handled. Static content can be served significantly faster than dynamic content,^[1] this makes for a major reduction in server load and allows for many more page views on modest hardware, lessening the budget for running the website.

Now, most Shitsaba installations are used by about ten people, so this is all irrelevant in those cases, but for larger boards such as 4chan, generating pages ahead of time makes a tremendous difference, and removes the need for typically flaky and awkward (and overly ENTERPRISE) "solutions" such as stuffing a caching proxy in front of the server. If the affected files are simply regenerated when they're changed, there's no need to constantly fuss with revalidating or invalidating the cached copies, and the problem of multiple requests causing simultaneous cache misses never even occurs.

<sub><sup>References
1. http://arnisoft.com/239/server-benchmark-apache-nginx-cherokee/</sup></sub>

JACKSON 5 GET!

PHP and some sort of database is pretty much all you'll need to get started. Finish a basic PHP tutorial and you should be able to do all the basics without breaking a sweat.

>>6
PHP sucks.

>>7
Listen here, jerkface. PHP does not Fuck off, ``faggot''

>>7
Every goddamn hosting provider in the world lets you write shitty PHP code hooked up to a shitty MySQL database.  Why tell OP to use a nice language and working database when they'd fail either way?

>>1
This board is for programmers.  If you don't program, read your SICP and come back here later.

>>8
Listen?  This is a text medium; I am reading, not listening.

>>9
I cannot let PHP be mentioned without a negative comment

>>10
I like the kind of teacher who encourages students to go hurt themselves.


Novice:  How do I write an 4chan?
Teacher: Use PHP and MySQL.
    (days pass)
Novice:  I have made an 4chan, but writing PHP is like
         having teeth pulled rectally, and the MySQL database
         supports transactions only half as well as my dog
         after he ate two pounds of butter.
Teacher: That is why you should have used Java and a
         NoSQL key-value store such as Cassandra or Voldemort.
    (weeks pass)
Novice:  I have remade an 4chan, but the Java code throws
         NullPointerExceptions like mad, the GC requires
         constant tuning, and the key-value store responds
         to queries like my younger brother when he's OD'd
         on Ritalin.
Teacher: You should use an agile design methodology and
         reimplement the front-end in Ruby on Rails.
    (days pass)
Novice:  Again, I have made an 4chan but...
Teacher: That is why you should reimplement it in Haskell,
         using the type system to automatically...
    (Novice tries to deck the Teacher, but the Teacher ducks)
Novice:  The type system will fuck up the schema upgrades
         you twat!
    (Novice lunges at Teacher, and is knocked to the floor
     by an errant broom.)
Teacher: Or perhaps you should consider using Common Lisp,
         defining macros for...
    (Novice tries to sweep kick Teacher, but Teacher jumps
     and lands on the Novice's foot.  There is an audible
     crack)
Novice:  Bastard! Library support for Common Lisp is
         scattered, and support for sockets is...
Teacher: Ah! You have learned well! Now tell me, how is it
         that 4chan runs now?
Novice:  Aren't they just running code based on existing
         imageboard software?
Teacher: Mm... yes, but what does that mean?
Novice:  It means that the code already works?
Teacher: And?
Novice:  And what?
Teacher: While clunky, over the years, the software has been
         fine tuned to perform exactly the task which has
         been asked of it.  Null pointer exceptions,
         problems with transactions, GC problems... these
         problems have been minimized by years of program
         tweaks and administrator experience.  There is no
         foolproof way to build a dynamic website but to
         use the technologies you know and through years of
         toil, mitigate the deficiencies in your technology
         one by one.

         The job of the Architect is to predict which
         technologies and techniques will work together
         to create a complete, elegant system.  This is a
         job that cannot be taught.  It is much coveted,
         but the path to it lies only through bitter
         experience, and long nights integrating Microsoft
         Excel with the company's J2EE backend.

Novice:  So, PHP, huh?
Teacher: PHP is good enough for Facebook, after all.

>>11
But PHP isn't good enough for Facebook; that's why they developed their own PHP to C++ compiler...

Using a better language/platform would likely result in less servers, less electricity use, and less pollution.  Avoiding PHP is the green thing to do.

>>12
So, by your logic, C isn't good enough for anyone because everyone uses a C-to-assembler compiler?

For low popularity apps, PHP is greener than e.g. C.  If you want to host a web app in C, you pretty much need your own VM slice, which is a big chunk of computer right there.  For PHP, you can get loaded on a shared box with 50 or 100 other PHP websites because you don't even blip on the CPU usage meter.

>>13
no, that's not what I said.  C and PHP are not the only languages.  there are web development language/environments that are more efficient than PHP in development and execution time.

and no one gives a shit about tiny web sites.

>>14
Some tiny web site is exactly what the OP was asking about.  Premature optimization, really.  Most CPUs on web servers are underutilized, and CPU only affects latency very slightly.  You can switch from PHP or Ruby on Rails to something more efficient like ASP .NET or J2EE or whatever, but all you'll notice in terms of performance is that (1) the web server still spends most of its time waiting for the database to respond, (2) the cpu usage meter has gone down from 10% to 5%, and (3) maybe, maybe, if you do enough trials you'll see that the response time has gone down a couple milliseconds.

An image board is almost all I/O, so optimize that FIRST (plug in a caching reverse proxy like Varnish) before you optimize CPU usage on the web server.  Reducing I/O with the reverse proxy will give you real power savings by reducing disk utilization.

The original argument was that using a better language or platform would be less servers, less electricity, and less pollution.  But in most cases it wouldn't make an impact, because most web sites only run on one web server to begin with.

PHP is shitty, but it's here now and you can get a PHP site hosted anywhere no problem.  You want to go something better like J2EE (don't laugh...) and you'll have to pay more or live with a SHARED HEAP, which is just fucking brilliant.

>>13
For low popularity apps, PHP is greener than e.g. C.  If you want to host a web app in C, you pretty much need your own VM slice, which is a big chunk of computer right there.
What on earth is that supposed to mean? C (i.e. a compiled binary) is supported virtually everywhere.

>>15
caching reverse proxy
Read >>4 again.

Aside from that, the tiny websites that no one cares about are not worth optimizing anything for. You could shit out the worst, most security-flaw-laden code in the world, and chances are very good nobody will ever notice or care.

It's when that one-in-a-hundred-thousand site actually stumbles by sheer accident on the right combination of timing, innovation, and a good initial userbase, and popularity starts going up, that you really need to put any of this into consideration at all. Before that, don't waste your time.

And as a final note, no website ever materialized out of nowhere with a strong userbase. Everything came from somewhere else; the "if you build it they will come" mentality that plagues these very same shitty 4chan-clone imageboards is pathetic. You create a site because you already have users who are interested in it, not the other way around.

>>17
My point was that if you come to /prog/ asking how to write an image board, the answer "go write it in PHP and SQL" is the correct answer for the situation.  If it gets a little popular, then putting Varnish in front will fix issues with the load with relatively minimal implementation effort.  As for security issues, there are bots out there smart enough to sniff SQL injection attacks, so changing your language doesn't do you much good (unless you use an ORM, which I tend to hate even more than PHP).

Pregenerating HTML in a static directory is, yes, better but I suspect there are some nontrivial synchronization issues involving simultaneous posts.  Such issues are far out of a newbie's league.

If you're the kind of person who has the programming skills to set up something that can handle a load like 4chan's, then you wouldn't be asking "how do I make an image board?"  Instead, you'd only stick around on /prog/ to troll.

Like what I'm doing. not that anyone cares

>>18
You're a fucking moron.

>>18
Pregenerating HTML in a static directory is, yes, better
What is this, 2001?

Thanks dudes but really, i know PHP and MySql and all those things like windows and ms paint.

I already did couple of things but I just asked you if making it normal way is good choice. Thanks for this big  discussion ;))

>>23
;))
Why are you so fat.

>>24
I actually lol'd. Thanks, I needed it.

>>22
Explain to me how hitting the database and recreating exactly the same content repeatedly, slowing down the speed at which pages can be served by at least two orders of magnitude, is in any way smarter or better than writing out an html file and then letting the web server serve static files, a very simple task, one which over many years has been carefully studied and highly optimized much more than anything else in the same problem domain.

If you're just going to be putting a caching proxy in front of it anyway, why not just open an html file and cut the middleman. You're outputting precisely the same data, no matter how or when you generate it; the only difference from the perspective of your script is where you send it -- to a file, or directly to the client. Except, in the latter case, you're also adding another process and pointless complexities of having to maintain a separate caching system when a filesystem would work perfectly fine and with ultimately less effort and higher efficiency.

>>26
Because that's the ``new'' way of doing it and according to >>22, newer is always better. You don't want to be using technology from back in 2001, do you?

>>27
In 2001 it was still completely possible to serve a fully dynamic site. It's not new, it's just dumb.

>>26
why not just open an html file and cut the middleman
Because of exactly the reason >>18 mentioned. There are non-trivial synchronization issues when multiple connections try to update a file, and there are issues with getting your static server to actually update when these files are replaced.

Letting the web server serve static files is actually not very well optimized for high loads. If you need to serve static content in an efficient way, you need a proxy webserver for your static content; i.e. you use a lighttpd server as the frontend which serves anything static, and any connections to a script fall through to a heavyweight Apache server to generate the content. So you have a middle-man anyway.

Might as well just use memcached; it's going to be a lot faster than proxying connections and fucking around with the filesystem, and it will be a heck of a lot easier to change your code to use memcached instead of the filesystem.

But I still agree with >>18 though, this is far outside the complexity of the vast majority of applications. I would just write it with Python/Postgres and forget about caching until it actually becomes a problem.

>>21
</h1>

1. Write an application that waits for new ``update-the-HTML-pages'' requests and executes them until they are nonexistent. Limit the request queue according to the available memory

2. Write a (layered) (compiled) application that does all the post validation mumbojumbo, persists the data and then invokes application #1

3. Write some JavaScript to lazyload the images in each board page

4. Serve the files generated by application #1 in a lighttpd server

5. Serve application #2 in an Apache server

6. ???

7. Cheap, fast 4chons

What's the problem with this approach?

>>31
When generating the index pages, you practically have to fetch every post in order to sort and maintain thread order, count omitted files and so on. You might as well rape the DB once per post instead of continually whenever a 404 occurs.

Also, if you do it strictly the way you described, it's quite easy to do something subtly wrong, such that you end up with lingering thread pages for old threads that were rolled off the board.

On top of that, if you use a strict queue, you're presumably not generating the pages right as posts are made, so there's a possibility that someone will post a new thread and then find it 404'd, unless I've misunderstood your description. If you're doing this on demand, you'd be better off writing to an html file and then using a sendfile()-like method directly afterward to send it to the client. A work queue is not the right way to go about this.

(And Javascript doesn't make any sense whatsoever)

>>33
Read SICP.

>>32
When generating the index pages, you practically have to fetch every post in order to sort and maintain thread order
What's the big deal? See >>4. Also, lrn2sql. If you have a smart database model you could keep fetches to the bare minimum.

Also, if you do it strictly the way you described, it's quite easy to do something subtly wrong, such that you end up with lingering thread pages for old threads that were rolled off the board.
It's never too hard to do something subtly wrong. Thankfully, we are EXPERT programmers.

On top of that, if you use a strict queue, you're presumably not generating the pages right as posts are made
Isn't that inevitable in a high-traffic website? How will the server keep up with hundreds of requests per minute?

so there's a possibility that someone will post a new thread and then find it 404'd
Welcome to 4chan.

If you're doing this on demand, you'd be better off writing to an html file and then using a sendfile()-like method directly afterward to send it to the client.
Yes, this is a good idea.

A work queue is not the right way to go about this.
Perhaps not, but I don't see a better alternative. Sometimes we have to settle for the lesser of evils; see capitalism.

>>31
Other than the technical problems, and the fact that it's obvious that you never wrote a web app with an architecture anything like that, the problem is that it doesn't exist.  Go build it and come back and maybe people won't think you're such a progtard.

Seriously.  Go try to build it.  It sounds like you need the experience.  Do you want to be yet another bastard talking out of his ass about things he's not qualified for?

(Among other technical problems, lighttpd has a slow memory leak, which is why people who were using that are switching, e.g., to nginx.)