FireflyFOSDEM PGDay 20162016-01-292016-01-313https://www.postgresql.eu/events/fosdem2016/schedule/08:3000:30OtherRegistrationhttps://www.postgresql.eu/events/fosdem2016/schedule/session/1194/09:0000:15OtherWelcome and openinghttps://www.postgresql.eu/events/fosdem2016/schedule/session/1193/Magnus Hagander09:2000:50HotelPostgreSQL High Availability, 2016 EditionNo one wants PostgreSQL to fail, but failure is just a part of life. There are lots of ways of getting high availability out of PostgreSQL, and the landscape is always changing.
We'll discuss the current situation for practical, reliable high-availability in PostgreSQL, including newly-emerging tools.https://www.postgresql.eu/events/fosdem2016/schedule/session/1144/Christophe Pettus10:2000:50HotelPostgreSQL Performance on EXT4, XFS, F2FS, BTRFS and ZFSOne of the most common question I'm asked by users and customer about configuring a Linux-based system for PostgreSQL is "What file fystem should I use to get the best performance?" Sadly, most of the recommendations is based on obsolete "common knowledge" passed from generation to generation. But in recent years the file systems improved a lot - we've seen both evolution of established file systems (EXT family, XFS, ...) and revolution in the form of BTRFS, ZFS or F2FS. And of course new kinds of hardware (SSDs) which certainly impacts the design of a file system.
I'm aware of benchmarks presented on Phoronix, but I consider them "informational" rather than a sound source of reliable data - I think we need much better benchmarks, so I've performed an extensive set of PostgreSQL benchmarks on a variety of file systems, with different configurations, etc.
This talk will present and discuss the results, and hopefully give some recommendations on which file system to use and how to configure it for PostgreSQL (and possibly for other applications).https://www.postgresql.eu/events/fosdem2016/schedule/session/1186/Tomas Vondra11:1000:20OtherCoffeehttps://www.postgresql.eu/events/fosdem2016/schedule/session/1195/11:3000:50HotelDeciphering two phase commitThe need to scale out PostgreSQL is a reality, thanks to our hunger for data. Ensuring ACID properties in a distributed system is both exciting as well as challenging. Two phase commit protocol ensures consistency and durability of transactions that distribute across multiple nodes. The ongoing extensible transaction manager API work, several massively parallel systems built off PostgreSQL are some instances of implementing the protocol in PostgreSQL. In this talk we explain working of two phase commit in the context of PostgreSQL. How costly DDL/DML statements become, is there an opportunity to optimize this cost, what are the implications on crash recovery? These are some of the questions we aim to delve into.https://www.postgresql.eu/events/fosdem2016/schedule/session/1177/Ashwin AgrawalAsim Praveen12:3000:50HotelManaging PostgreSQL with AnsibleThis talk will cover the topics listed below:
General info about configuration management and benefits of using configuration management tools
Ansible overview and advantages of Ansible comparing to other configuration management tools
Basics of Ansible such as roles, tasks, handlers, modules, playbooks etc
Advanced functionalites of Ansible and more scenarios
Info about Ansible PostgreSQL modules and their usage with examples
A playbook example using PostgreSQL modules and managing PostgreSQL instances on AWS.https://www.postgresql.eu/events/fosdem2016/schedule/session/1164/Gülçin Yıldırım Jelinek13:2001:00OtherLunchhttps://www.postgresql.eu/events/fosdem2016/schedule/session/1196/14:2000:50HotelHighly available PostgreSQL with PatroniAutomatic failover is hard, but sometimes there is no other choice. Zalando, having hundreds of autonomous teams and more than 150 database clusters, was exactly in this situation. To tackle this problem we’ve developed Patroni, a core of our HA solution. It is based on the idea of combining a distributed configuration system with PostgreSQL in order to reliably detect master failures and guard against split brain and other common failure issues. Patroni is based on Compose Governor, although it is a significant improvement: it is stable, more performant and has numerous new features, such as support for Zookeeper, ability to perform on-demand failovers, creation of new replicas with methods other than pg_basebackup, multi-threading to perform long-running tasks, support for pg_rewind and other improvements. Patroni is stable and runs new production databases inside Zalando. It’s also an open-source project, actively developed in the open on github: https://github.com/zalando/patroni
This talk will describe the ideas behind Patroni, architecture and use-cases, from running it on developer’s laptop for testing to managing numerous database clusters in the cloud.https://www.postgresql.eu/events/fosdem2016/schedule/session/1141/Oleksii Kliukin15:2000:50HotelLogical decoding for applications: Solr full-text search in near-real time with pglogicalLogical decoding in PostgreSQL 9.4+ was added to support logical replication - but you can use also it to supply a continuous database change stream to external applications.
With the pglogical_output plugin this is now approachable for people who aren't experts in the PostgreSQL backend and writing extensions. You don't even need to know C.
Feed database changes into a business intelligence system, send them to a message queue, stream them to a write-once secure audit log, send them to an intrusion detection system... wherever you need a change stream from your database you can use logical decoding.
As an example I will use the new pglogical_output plugin and psycopg2's coming support for the logical replication protocol to feed changes from a selection of tables to Apache Solr for near-real-time full-text indexing and search.https://www.postgresql.eu/events/fosdem2016/schedule/session/1182/Craig Ringer16:1000:20OtherCoffeehttps://www.postgresql.eu/events/fosdem2016/schedule/session/1197/16:3000:50HotelStreaming huge databases using logical decodingIn this talk, I will discuss Zalando's open-source data collection prototype that uses PostgreSQL's logical replication streaming capabilities to collect data from the database and stream it for later processing (Data Lake, Operational Data Store, automatic process monitoring, KPI calculation systems). The process includes taking a consistent snapshot of the database in a distributed and efficient manner and streaming the subsequent changes via a logical replication slot.
This approach can be used for online PostgreSQL major version upgrades, to fill event queues, for partial replication, etc. Use of Python makes it easy to support different target systems and to prototype rapidly.
The audience will come away with new ideas for how to use PostgreSQL logical decoding in daily business.
https://www.postgresql.eu/events/fosdem2016/schedule/session/1187/Oleksandr Shulgin11:0000:50H.1309 (Van Rijn)Unified Framework for Big Data Foreign Data Wrappers (FDW)With the introduction of Foreign Data Wrappers in Postgres 9.1, access to distributed systems such as Hdfs, HBase, Hive with their multiple data formats is feasible.
However, the existing FDW implementations for Big Data systems, such as Hdfs or Hive, lack a few key features and doesn’t have a common framework.
PXF provides a unified extensible framework for accessing any distributed system data source. Existing plugins include loading and querying of data stored in HDFS, HBase and Hive. It supports a wide range of data formats such as Text, Avro, Sequence, Hive RCFile, ORC, Parquet and Avro formats and HBase. The pluggable framework makes it very convenient for adding new custom plugins. It also supports advanced statistics and filter pushdown.
PXF is an open source project and is currently being used by Apache HAWQ’s external table via PXF’s exposed REST API and is in the process of being integrated with other SQL engines.
With the integration of PXF into Postgres FDW, we can achieve a single unified pluggable framework to read and write any distributed system data source. PXF also abstracts Postgres from any remote client dependencies and provides a clean installation mechanism.
https://www.postgresql.eu/events/fosdem2016/schedule/session/1176/Shivram Mani12:0000:50H.1309 (Van Rijn)Improving Postgres' Buffer ManagerPostgresql's buffer manager has parts where it's showing its age. We'll discuss how it currently works, what problems there are, and what attempts are in progress to rectify these weaknesses:
* Lookups in the buffer cache are expensive
* The buffer mapping table is organized as a hash table, which makes efficient implementations of prefetching, write coalescing, dropping of cache contents hard
* Relation extension scales badly
* Cache replacement is inefficient
* Cache replacement replaces the wrong buffers https://www.postgresql.eu/events/fosdem2016/schedule/session/1184/Andres Freund13:0000:50H.1309 (Van Rijn)JSON By ExampleA deep walk through PostgreSQL JSON features with data examples. This does include the new and shiny PostgreSQL 9.5 JSON features.https://www.postgresql.eu/events/fosdem2016/schedule/session/1171/Stefanie Janine Stölting14:0000:50H.1309 (Van Rijn)Explaining the Postgres Query OptimizerThe optimizer is the "brain" of the database, interpreting SQL queries and determining the fastest method of execution. This talk uses the explain command to show how the optimizer interprets queries and determines optimal execution. The talk will assist developers and administrators in understanding how Postgres optimally executes their queries and what steps they can take to understand and perhaps improve its behavior.https://www.postgresql.eu/events/fosdem2016/schedule/session/1161/Bruce Momjian15:0000:50H.1309 (Van Rijn)UPSERT use casesPostgreSQL 9.5 will have support for a feature that is popularly known as "UPSERT" - the ability to either insert or update a row according to whether an existing row with the same key exists. If such a row already exists, the implementation should update it. If not, a new row should be inserted. This is supported by way of a new high level syntax (a clause that extends the INSERT statement) that more or less relieves the application developer from having to give any thought to race conditions. This common operation for client applications is set to become far simpler and far less error-prone than legacy ad-hoc approaches to UPSERT involving subtransactions. Moreover, the new implementation performs much better than those legacy approaches.
While the feature is most obviously compelling for OLTP and web application use cases, it's also true that the syntax is powerful enough to be very useful in many real world data integration scenarios. The non-standard PostgreSQL syntax offer explicit, fine grained control over where and how to update. For example, an update may not actually affect an existing row due to not satisfying some additional criteria (i.e. due to not passing the ON CONFLICT ... DO UPDATE special, dedicated WHERE clause).
This talk gives an overview of the feature from a high level, and examines these use cases. You will learn how you might want to use the new UPSERT feature in your application beyond the obvious. In passing, there will be brief discussion of why UPSERT's implementation proved to be a hard problem, and, relatedly, why a custom syntax was used instead of the SQL standard's MERGE syntax.https://www.postgresql.eu/events/fosdem2016/schedule/session/1169/Peter Geoghegan16:0000:50H.1309 (Van Rijn)Using PostgreSQL for Bibliographic DataAn investigation of how PostgreSQL and its lastest capabilities (JSONB data type, GIN indices, Full Text Search) can be used to store, index and perform queries on structured Bibliographic Data such as MARC21/MARCXML, breaking the dependence on proprietary and arcane or obsolete software products.https://www.postgresql.eu/events/fosdem2016/schedule/session/1183/Jimmy Angelakos