Parallel execution of SQL

2009-01-02 19:10:45 +0100

The end of increasing clockspeeds

The clock-speed for processors doesn't increase anymore due to physical limits. The only way for processor-makers like Intel and AMD to increase the speed of a processor is to add extra cores. Making software scale near-lineair in relation to the number of cores (meaning: doubling the number of cores should result in a doubling of the performance of a program) is quite hard with the current way of programming done by most programmers/architects.

Parallel execution
Fortunately there's one language which is very well design to be executed in parallel querying databases using SQL.  Running queries in parallel means the query is split up in subqeuries. Each of those queries is run in parallel and then combined.
For parallelism to have to have advantage, there has to be either a database server having multiple cores or CPU's or the database has to run distributed on multiple nodes (servers/computers). In the latter case, when running on multiple nodes, there's has to be a very fast network connecting these nodes. Also the database-server software has to support running queries in parallel. Well-known database-software such as Oracle can do this out-of-the-box.

When you have an application that is used by a lot of users doing much of the same at the same time - more than the number of nodes and cores-per server - then running queries in parallel doesn't have any advantage.  However, when a small number (or just one) users are running heavy queries - less than the number of nodes and cores-per-server - running queries in parallel can have great advantages.

A SQL statement like SELECT * FROM customer, and especially aggregation functions like SELECT avg(grade) FROM students or SELECT max(count(*)) FROM student GROUP BY class. Inserts can run parallel too, although that'll only have a slight advantage when inserting a lot of data at the same time.

Modern DBMS (Database Management Software, usually called database-software or simply databases) can run queries parallel on multi-core systems out-of-the box, like the well known DBMS of Oracle. Running queries on multiple-nodes with software such as Oracle RAC. For a detailed explanation on parallel execution in Oracle see the whitepaper Oracle SQL Parallel Execution , or a posting of Don Burleson for a short explanation.

In short, a modern DBMS can do a lot more then you may imagine. Smart SQL can solve quite some problems in concurent programming and optimazing software for multi-core or multi-node environments.

Read more

Devoxx'08

2008-12-24 12:52:08 +0100

This posting is a translation of the posting on my old weblog Devoxx'08 in Dutch.

Of all the presentations that I've seen, this is my top:

  1. From Concurrent to Parallel by Brian Goetz - A very good presentation. The Parallel Array structure and the Fork-Join - both in Java 7 - seem very powerful.
    I bought the by Brian Goetz
    Java Concurrency in practice. Unfortunately treated the new feature of Java 7 does not, the book is from 2006. However, for a good book shouldn't outdate that soon, so I don't think that should be a problem.
  2. The Feel of Scala by Bill Venners - An extraordinary promising and good language.
    I especially liked the demonstration how Scala has a way to extend functionality of existing classes in a similar way that is possible in Ruby (by addin methods, extending the language), but with static typing.
    Since Scala is statistically typed, refactoring can be deterministic. This means your code is guarentees to work the same, after refactoring. Dynamically typed languages only support heuristic refactoring. This means heuristics and unit-tests are needed during and after refactoring.
    There are also many other very nice features which makes Scala, for which this posting is too short to list all.
    The newly purchased book
    Programming in Scala is now on my desk.
  3. Be smart! by Ivar Jacobson - A very energetic man. I agree with what he told, and not all that he told was that new, but he can bring his point very well.
  4. RESTful Design, Patterns and Anti-Patterns by Stefan Tilkov - Through his talk, I became very enthusiastic about REST. Hearing SOAP is actually a bloated (allthough he did not stated that as-such) remoting protocol which doesn't add that much to other remoting technologies is refreshing to hear.
    Using REST, the WWW is much better utilizied. REST brings what SOAP used to promise: a the web as a set of connecting services.
    He presented a few patterns and anti-patterns in using REST.
  5. Effective Pairing: The Good, the Bad and the Ugly by Dave Nicolette - Fun and inspiring. Examples how to do pairing (in programming) and especially how not to. Quite interesting, allthough I think the presentation is less interesting if you're not there directly.
  6. Implementing agile at enterprise level by Jef Cumps - Quite informative demonstration how Scrum was introduced and is used at The Belgian Post. I liked the picture of a real Scrum-board as they used it, with yellow and pink-post-its.
    Scrum can be introduced in large organizations as he demonstrated, if the benefit is clear to all parties.
  7. What's new in Spring Framework 3.0? by Arjen Poutsma and Alef Arendsen - Spring remains a powerful framework if you use Java. Very interesting to hear what new features Spring 3.0 brings, since I use the framework almost everyday in my work.
    In Spring 3, finally Spring is going to use Java 1.5 features such as generics in the core, which is already the case for the modules.
    They demonstrated the new REST-features of Spring 3 (this talk was after the talk on REST). As they demonstrated, using Spring it'll be easy to set up or consume a REST service using a few simple annotations and configuration.
  8. Java + XSLT 2.0
    Doug Tidwell - - Pretty interesting, and good speaker.
    I was especially interested, because I recently wrote a short article on XSLT.
    XSLT 2.0 seems very useful, especially the ability to call Java from XSLT.
  9. Patterns in Mule
    by Antoine Borg Quite interesting, Mule seems a very nice tool. I've worked in organizations in the past, where a ESB seems to fit very nicely. Mule, being open-source and seemingly easy to use could be introduced easily.
    Only thing was the speaker was not that good. Sometimes he appeared just talking as if he was reading aloud.
  10. How Do We Test This Stuff?
    by Frank Cohen - Of course I wish to test well. The speaker spoke good, but the content was not so interesting, I heard not much news. Especially on open-source test tools, PushToTest, Selenium, SoapUI.
  11. Preventing bugs with pluggable type-checking for Java
    by Mahmood Ali - Very promising and interesting idea and framework: a pluggable type system via Annotations. Only I had the feeling that the speaker had a cold and was a little nervous.
  12. Introduction to NIO.2
    by Jean-Francois Arcand - This presentation was not as good as I hoped.The speaker spoke messy, half-finished sentences, and with a heavy French accent. NIO2 and Grizzly framework are powerful technologies, I got that though.

There were other presentations that were very good also, I heard of others. I look forward to when the presentations online at Parleys.

Devoxx logo


Java Concurrency In Practice
Java Concurrency In Practice
Tim Peierls & Brian Goetz

Professional Java Development With The Spring Framework
Professional Java Development With The Spring Framework
Rod Johnson & Juergen Hoeller

XSLT
XSLT
Doug Tidwell

Java Nio
Java Nio
Ron Hitchens

Read more

Archive

subscribe via RSS