Skip to content

L
lsm_tree_postgres
Commit 67035927 authored by Paras Garg's avatar Paras Garg
Browse files
Options

readme

parent 0acbed60
Expand all Show whitespace changes
Inline Side-by-side
Showing with 19 additions and 1241 deletions
README.md 0 → 100644
View file @ 67035927
LSM tree implemented using standard Postgres B-Tree indexes.
Two levels of index are L0 and L1.
LO is used to store initial inserts and L1 is used.
FOR REMAINING POINT AND INSTALLATION
see the report
\ No newline at end of file
diff.txt 0 → 100644
View file @ 67035927
select tablename ,indexname, indexdef from pg_indexes where tablename='t';
logfile deleted 100644 → 0
View file @ 0acbed60
2020-12-14 01:01:11.215 IST [53734] FATAL: lock file "postmaster.pid" already exists
2020-12-14 01:01:11.215 IST [53734] HINT: Is another postmaster (PID 2917) running in data directory "/home/paras/postgres13/install/data"?
2020-12-14 01:16:10.828 IST [54122] LOG: starting PostgreSQL 13.0 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0, 64-bit
2020-12-14 01:16:10.828 IST [54122] LOG: listening on IPv4 address "127.0.0.1", port 5432
2020-12-14 01:16:10.926 IST [54122] LOG: listening on Unix socket "/tmp/.s.PGSQL.5432"
2020-12-14 01:16:11.072 IST [54123] LOG: database system was shut down at 2020-12-14 01:16:07 IST
2020-12-14 01:16:11.152 IST [54122] LOG: database system is ready to accept connections
2020-12-14 01:18:04.122 IST [54131] ERROR: syntax error at or near "enalyse" at character 1
2020-12-14 01:18:04.122 IST [54131] STATEMENT: enalyse
;
2020-12-14 01:18:11.225 IST [54131] ERROR: syntax error at or near ";" at character 9
2020-12-14 01:18:11.225 IST [54131] STATEMENT: EXPLAIN ;
2020-12-14 01:19:36.421 IST [54122] LOG: server process (PID 54131) was terminated by signal 11: Segmentation fault
2020-12-14 01:19:36.421 IST [54122] DETAIL: Failed process was running: select * from t where k=3;
2020-12-14 01:19:36.421 IST [54122] LOG: terminating any other active server processes
2020-12-14 01:19:36.421 IST [54127] WARNING: terminating connection because of crash of another server process
2020-12-14 01:19:36.421 IST [54127] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2020-12-14 01:19:36.421 IST [54127] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2020-12-14 01:19:36.423 IST [54162] FATAL: the database system is in recovery mode
2020-12-14 01:19:36.424 IST [54122] LOG: all server processes terminated; reinitializing
2020-12-14 01:19:36.534 IST [54163] LOG: database system was interrupted; last known up at 2020-12-14 01:16:11 IST
2020-12-14 01:19:36.806 IST [54163] LOG: database system was not properly shut down; automatic recovery in progress
2020-12-14 01:19:36.868 IST [54163] LOG: redo starts at 0/1C6CFDC0
2020-12-14 01:19:36.887 IST [54163] LOG: invalid record length at 0/1C6D1D90: wanted 24, got 0
2020-12-14 01:19:36.887 IST [54163] LOG: redo done at 0/1C6D1D58
2020-12-14 01:19:37.359 IST [54122] LOG: database system is ready to accept connections
2020-12-13 22:10:51.822 IST [2998] LOG: starting PostgreSQL 13.0 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0, 64-bit
2020-12-13 22:10:51.822 IST [2998] LOG: listening on IPv4 address "127.0.0.1", port 5432
2020-12-13 22:10:51.941 IST [2998] LOG: listening on Unix socket "/tmp/.s.PGSQL.5432"
2020-12-13 22:10:52.142 IST [3001] LOG: database system was interrupted; last known up at 2020-12-14 01:21:27 IST
2020-12-13 22:10:52.589 IST [3001] LOG: database system was not properly shut down; automatic recovery in progress
2020-12-13 22:10:52.667 IST [3001] LOG: redo starts at 0/1C6D1F68
2020-12-13 22:10:52.696 IST [3001] LOG: invalid record length at 0/1C6F5968: wanted 24, got 0
2020-12-13 22:10:52.696 IST [3001] LOG: redo done at 0/1C6F5930
2020-12-13 22:10:53.258 IST [2998] LOG: database system is ready to accept connections
2020-12-13 22:11:30.607 IST [3010] ERROR: type "k" does not exist at character 20
2020-12-13 22:11:30.607 IST [3010] STATEMENT: create table t(int k,int v);
2020-12-13 22:14:39.402 IST [3010] ERROR: relation "lsm_handler" does not exist at character 15
2020-12-13 22:14:39.402 IST [3010] STATEMENT: select * from lsm_handler;
2020-12-14 12:09:49.954 IST [2998] LOG: received fast shutdown request
2020-12-14 12:09:50.047 IST [2998] LOG: aborting any active transactions
2020-12-14 12:09:50.049 IST [2998] LOG: background worker "logical replication launcher" (PID 3007) exited with exit code 1
2020-12-14 12:09:50.050 IST [3002] LOG: shutting down
2020-12-14 12:09:50.266 IST [2998] LOG: database system is shut down
postgres: invalid argument: "log_min_messages=debug4"
Try "postgres --help" for more information.
2020-12-14 12:11:48.309 IST [6452] LOG: starting PostgreSQL 13.0 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0, 64-bit
2020-12-14 12:11:48.309 IST [6452] LOG: listening on IPv4 address "127.0.0.1", port 5432
2020-12-14 12:11:48.394 IST [6452] LOG: listening on Unix socket "/tmp/.s.PGSQL.5432"
2020-12-14 12:11:48.551 IST [6453] LOG: database system was shut down at 2020-12-14 12:09:50 IST
2020-12-14 12:11:48.631 IST [6452] LOG: database system is ready to accept connections
2020-12-14 12:11:55.745 IST [6462] ERROR: syntax error at or near "-" at character 1
2020-12-14 12:11:55.745 IST [6462] STATEMENT: -d
;
2020-12-14 12:12:01.861 IST [6462] ERROR: syntax error at or near "-" at character 1
2020-12-14 12:12:01.861 IST [6462] STATEMENT: -d
;
2020-12-14 12:12:06.535 IST [6462] ERROR: syntax error at or near "d" at character 1
2020-12-14 12:12:06.535 IST [6462] STATEMENT: d;
2020-12-14 12:12:13.264 IST [6462] ERROR: syntax error at or near "debug" at character 1
2020-12-14 12:12:13.264 IST [6462] STATEMENT: debug
;
2020-12-14 12:12:19.468 IST [6462] ERROR: syntax error at or near "debug" at character 1
2020-12-14 12:12:19.468 IST [6462] STATEMENT: debug-level;
2020-12-14 12:20:35.554 IST [6452] LOG: received fast shutdown request
2020-12-14 12:20:35.606 IST [6452] LOG: aborting any active transactions
2020-12-14 12:20:35.609 IST [6452] LOG: background worker "logical replication launcher" (PID 6459) exited with exit code 1
2020-12-14 12:20:35.609 IST [6454] LOG: shutting down
2020-12-14 12:20:36.171 IST [6452] LOG: database system is shut down
lsm (copy).c deleted 100644 → 0
View file @ 0acbed60
This diff is collapsed.
lsm (copy).h deleted 100644 → 0
View file @ 0acbed60
/*
* It is too expensive to check index size at each insert because it requires traverse of all index file segments and calling lseek for each.
* But we do not need precise size, so it is enough to do it at each n-th insert. The lagest B-Tree key size is abut 2kb,
* so with N=64K in the worst case error will be less than 128Mb and for 32-bit key just 1Mb.
*/
#define LSM3_CHECK_TOP_INDEX_SIZE_PERIOD (64*1024) /* should be power of two */
/*
* Control structure for Lsm3 index located in shared memory
*/
typedef struct
{
Oid base; /* Oid of base index */
Oid heap; /* Oid of indexed relation */
Oid top[2]; /* Oids of two top indexes */
int access_count[2]; /* Access counter for top indexes */
int active_index; /* Index used for insert */
uint64 n_merges; /* Number of performed merges since database open */
uint64 n_inserts; /* Number of performed inserts since database open */
volatile bool start_merge; /* Start merging of top index with base index */
volatile bool merge_in_progress; /* Overflow of top index intiate merge process */
PGPROC* merger; /* Merger background worker */
Oid db_id; /* user ID (for background worker) */
Oid user_id; /* database Id (for background worker) */
int top_index_size; /* Size of top index */
slock_t spinlock; /* Spinlock to synchronize access */
} Lsm3DictEntry;
/*
* Opaque part of index scan descriptor
*/
typedef struct
{
Lsm3DictEntry* entry; /* Lsm3 control structure */
Relation top_index[2]; /* Opened top index relations */
SortSupport sortKeys; /* Context for comparing index tuples */
IndexScanDesc scan[3]; /* Scan descriptors for two top indexes and base index */
bool eof[3]; /* Indicators that end of index was reached */
bool unique; /* Whether index is "unique" and we can stop scan after locating first occurrence */
int curr_index; /* Index from which last tuple was selected (or -1 if none) */
} Lsm3ScanOpaque;
/* Lsm3 index options */
typedef struct
{
BTOptions nbt_opts; /* Standard B-Tree options */
int top_index_size; /* Size of top index (overrode lsm3.top_index_size GUC */
bool unique; /* Index may not contain duplicates. We prohibit unique constraint for Lsm3 index
* because it can not be enforced. But presence of this index option allows to optimize
* index lookup: if key is found in active top index, do not search other two indexes.
*/
} Lsm3Options;
lsm.c
View file @ 67035927
...@@ -85,7 +85,6 @@ lsm_truncate_index(Relation index, Oid heap_oid) ...@@ -85,7 +85,6 @@ lsm_truncate_index(Relation index, Oid heap_oid)
static void static void
lsm_merge_indexes(Oid dst_oid, Relation top_index, Oid heap_oid) lsm_merge_indexes(Oid dst_oid, Relation top_index, Oid heap_oid)
{ {
//Relation top_index = index_open(src_oid, AccessShareLock);
elog(NOTICE,"merge dest_oid %d,heap rel Oid %d",dst_oid,heap_oid); elog(NOTICE,"merge dest_oid %d,heap rel Oid %d",dst_oid,heap_oid);
Relation heap = table_open(heap_oid, AccessShareLock); Relation heap = table_open(heap_oid, AccessShareLock);
Relation base_index = index_open(dst_oid, RowExclusiveLock); Relation base_index = index_open(dst_oid, RowExclusiveLock);
...@@ -104,7 +103,7 @@ lsm_merge_indexes(Oid dst_oid, Relation top_index, Oid heap_oid) ...@@ -104,7 +103,7 @@ lsm_merge_indexes(Oid dst_oid, Relation top_index, Oid heap_oid)
IndexTuple itup = scan->xs_itup; IndexTuple itup = scan->xs_itup;
if (BTreeTupleIsPosting(itup)) if (BTreeTupleIsPosting(itup))
{ {
/* Some dirty coding here related with handling of posting items (index deduplication). /*
* If index tuple is posting item, we need to transfer it to normal index tuple. * If index tuple is posting item, we need to transfer it to normal index tuple.
* Posting list is representing by index tuple with INDEX_ALT_TID_MASK bit set in t_info and * Posting list is representing by index tuple with INDEX_ALT_TID_MASK bit set in t_info and
* BT_IS_POSTING bit in TID offset, following by array of TIDs. * BT_IS_POSTING bit in TID offset, following by array of TIDs.
...@@ -129,7 +128,6 @@ tes */ ...@@ -129,7 +128,6 @@ tes */
} }
index_endscan(scan); index_endscan(scan);
base_index->rd_rel->relam = save_am; base_index->rd_rel->relam = save_am;
// index_close(top_index, AccessShareLock);
index_close(base_index, RowExclusiveLock); index_close(base_index, RowExclusiveLock);
table_close(heap, AccessShareLock); table_close(heap, AccessShareLock);
} }
...@@ -214,6 +212,7 @@ lsm_create_l1_if_not_exits(Relation heap,Relation index,LsmMetaData* lsmMetaCopy ...@@ -214,6 +212,7 @@ lsm_create_l1_if_not_exits(Relation heap,Relation index,LsmMetaData* lsmMetaCopy
char* newName; char* newName;
char* l0name; char* l0name;
Relation l1; Relation l1;
if(lsmMetaCopy->l1==InvalidOid){
l0name=index->rd_rel->relname.data; l0name=index->rd_rel->relname.data;
newName=(char*)palloc(NAMEDATALEN+2); newName=(char*)palloc(NAMEDATALEN+2);
newName[0]='L'; newName[0]='L';
...@@ -221,7 +220,6 @@ lsm_create_l1_if_not_exits(Relation heap,Relation index,LsmMetaData* lsmMetaCopy ...@@ -221,7 +220,6 @@ lsm_create_l1_if_not_exits(Relation heap,Relation index,LsmMetaData* lsmMetaCopy
for(int i=0;i<NAMEDATALEN;++i){ for(int i=0;i<NAMEDATALEN;++i){
newName[i+2]=l0name[i]; newName[i+2]=l0name[i];
} }
if(lsmMetaCopy->l1==InvalidOid){
lsmMetaCopy->l1= index_concurrently_create_copy( lsmMetaCopy->l1= index_concurrently_create_copy(
heap,/*heap relation*/ heap,/*heap relation*/
index->rd_id,/*old oid*/ index->rd_id,/*old oid*/
......
lsm.conf deleted 100644 → 0
View file @ 0acbed60
shared_preload_libraries = 'lsm'
lsm.top_index_size=1MB
lsm.h
View file @ 67035927
...@@ -10,6 +10,10 @@ typedef struct ...@@ -10,6 +10,10 @@ typedef struct
int top_index_size; /* Size of top index */ int top_index_size; /* Size of top index */
} LsmMetaData; } LsmMetaData;
void
index_rebuild(Relation heapRelation,
Relation indexRelation,
IndexInfo *indexInfo);
void void
lsm_create_l1_if_not_exits( lsm_create_l1_if_not_exits(
......
lsm.o
View file @ 67035927
No preview for this file type
lsm.so
View file @ 67035927
No preview for this file type
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment