| Age | Commit message (Collapse) | Author | Files |
|---|
|
|
|
The implementation of _gdbm_cache_flush becomes prohibitively
inefficient during extensive updates of large databases. The
bug was reported at https://github.com/Perl/perl5/issues/19306.
To fix it, make sure that all changed cache entries are placed at
the head of the cache_mru list, forming a contiguous sequence.
This way a potentially long iteration over all cache entries can be
cut off at the first entry with ca_changed == FALSE.
This commit also gets rid of several superfluous fields in
struct gdbm_file_info:
- cache_entry
Not needed, because the most recently used cache entry
(cache_mru) is always the current one.
- bucket_changed
dbf->cache_mru->ca_changed reflects the status of the current
bucket.
- second_changed
Not needed because _gdbm_cache_flush, which flushes all changed
buckets, is now invoked unconditionally by _gdbm_end_update (and
also whenever dbf->cache_mru changes).
* src/gdbmdefs.h (struct gdbm_file_info): Remove cache_entry. The
current cache entry is cache_mru.
Remove bucket_changed, and second_changed.
All uses changed.
* src/proto.h (_gdbm_current_bucket_changed): New inline function.
* src/bucket.c (_gdbm_cache_flush): Assume all changed elements form
a contiguous sequence beginning with dbf->cache_mru.
(set_cache_entry): Remove. All callers changed.
(lru_link_elem,lru_unlink_elem): Update dbf->bucket as necessary.
(cache_lookup): If the obtained bucket is not changed and is going
to become current, flush all changed cache elements.
* src/update.c (_gdbm_end_update): Call _gdbm_cache_flush unconditionally.
* src/findkey.c: Use dbf->cache_mru instead of the removed dbf->cache_entry.
* src/gdbmseq.c: Likewise.
* tools/gdbmshell.c (_gdbm_print_bucket_cache): Likewise.
* src/falloc.c: Use _gdbm_current_bucket_changed to mark the current
bucket as changed.
* src/gdbmstore.c: Likewise.
* src/gdbmdelete.c: Likewise. Use _gdbm_current_bucket_changed.
* tests/gtcacheopt.c: Fix typo.
* tests/gtload.c: New option: -cachesize
|
|
|
|
|
|
This fixes https://puszcza.gnu.org.ua/bugs/index.php?502.
* src/findkey.c (gdbm_bucket_element_valid_p): Verify also if the
key/data pair ends at a valid file offset.
(_gdbm_read_entry): Set up the cache data only on success.
* src/gdbmdefs.h (gdbm_bucket_element_valid_p): Remove declaration.
(gdbm_file_info): New member 'file_size' keeps the recently retrieved
value of the file size. It is invalidated wherever data is written to
the database.
* src/fullio.c (_gdbm_full_write,_gdbm_file_extend): Invalidate
dbf->file_size.
* src/gdbmopen.c (gdbm_fd_open): Initialize file_size to -1.
(_gdbm_file_size): New function.
* src/mmap.c (_gdbm_file_size): Remove.
* src/proto.h (_gdbm_file_size): New proto.
|
|
|
|
|
|
* src/cachetree.c (_gdbm_node_set_elem): Remove function.
* src/findkey.c: Finalize efaefe4c.
|
|
* src/findkey.c (_gdbm_read_entry): Revert dc176a5c
|
|
The new bucket cache uses the least recently used replacement
policy (instead of the least recently read, implemented previously).
It also allows for quick bucket lookups by the corresponding
disk address. To this effect the cache entries form a red-black
tree sorted by bucket address.
Additionally, data buckets are also cached.
* README: Describe the new branch.
* src/bucket.c: Rewrite cache support.
* src/cachetree.c: New file.
* src/Makefile.am: Add new file.
* src/findkey.c (_gdbm_read_entry): Use _gdbm_fetch_data.
This ensures data pages are cached as well as buckets.
* src/gdbm.h.in (GDBM_BUCKET_CACHE_CORRUPTED): New error code.
(gdbm_cache_stat): New struct.
(gdbm_get_cache_stats): New proto.
* src/gdbmclose.c (gdbm_close): Call _gdbm_cache_free to dispose
of the cache.
* src/gdbmdefs.h (cache_elem_color): New data type.
(cache_elem): New members: ca_left, ca_right, ca_node, and
ca_hits.
(cache_tree): New typedef.
(gdbm_file_info): Remove bucket_cache and last_read.
New fields: cache_num, cache_tree, cache_mru, cache_lru,
cache_avail, cache_access_count.
* src/gdbmerrno.c: Handle GDBM_BUCKET_CACHE_CORRUPTED.
* src/gdbmopen.c (gdbm_fd_open): Change cache initialization.
(_gdbm_init_cache, _gdbm_cache_entry_invalidate: Remove.
* src/gdbmsetopt.c (setopt_gdbm_setcachesize): Cache can be
re-initialized on the fly.
* src/gdbmtool.c: Change bucket printing routines.
* src/proto.h (_gdbm_read_bucket_at): Remove.
(_gdbm_fetch_data,_gdbm_cache_init,_gdbm_cache_free)
(_gdbm_cache_flush,_gdbm_cache_elem_new)
(_gdbm_cache_tree_alloc,_gdbm_cache_tree_destroy)
(_gdbm_cache_tree_delete,_gdbm_rbt_remove_node)
(_gdbm_cache_tree_lookup): New protos.
(_gdbm_init_cache,_gdbm_cache_entry_invalidate): Remove.
* src/recover.c (_gdbm_finish_transfer): Adapt to the new
cache structure.
* src/update.c: Likewise.
* tests/setopt00.at: Fix second GDBM_SETCACHESIZE test.
|
|
|
|
* src/findkey.c (_gdbm_read_entry): Reallocate cache_entry->ca_data.dptr only
if necessary.
* src/gdbmdefs.h (data_cache_elem): New member: dsize
* src/gdbmdelete.c: Don't free cache_entry->ca_data.dptr
* src/gdbmopen.c (_gdbm_init_cache): Initialize ca_data.dsize
|
|
Rename: __read to gdbm_file_read
__write to gdbm_file_write
__lseek to gdbm_file_seek
__fsync to gdbm_file_sync
|
|
The hooks were introduced as a temporary tool in de7834e9. They did
their job and are not necessary any more.
|
|
* Makefile.am (set-dist-date): New rule
(dist-hook): Catch FIXMEs in NEWS.
* NEWS: Updated.
* src/findkey.c (gdbm_bucket_element_valid_p): New function.
(_gdbm_read_entry): Validate the retrieved bucket element.
* src/gdbm.h.in (gdbm_recovery): New member: duplicate_keys.
(GDBM_BAD_HASH_TABLE): New error code.
* src/gdbmdefs.h (TYPE_WIDTH,SIGNED_TYPE_MAXIMUM)
(OFF_T_MAX): New defines.
(off_t_sum_ok): New function.
(gdbm_bucket_element_valid_p): New prototype.
* src/gdbmerrno.c: Support for GDBM_BAD_HASH_TABLE code.
* src/gdbmtool.c (recover_handler): Fix argument counting.
New argument 'summary' prints statistics summary at the end
of the run.
(export_handler,import_handler): Fix argument counting.
* src/mmap.c (SUM_FILE_SIZE): Rewrite as inlined function.
Add error checking.
(_gdbm_mapped_remap): More error checking.
* src/recover.c (run_recovery): Don't bail out on GDBM_CANNOT_REPLACE.
(gdbm_recover): Initialize duplicate_keys
* src/systems.h: Include limits.h
|
|
|
|
|
|
|
|
Use the GDBM_SET_ERRNO and GDBM_SET_ERRNO2 macros to make
sure the error gets reported in debug output.
* src/fullio.c (_gdbm_full_read)
(_gdbm_full_write): Return -1 and set gdbm_errno
on error.
* src/bucket.c: Use GDBM_SET_ERRNO(2?) or
GDBM_DEBUG where necessary.
* src/falloc.c: Likewise.
* src/findkey.c: Likewise.
* src/gdbmdefs.h: Likewise.
* src/gdbmopen.c: Likewise.
* src/gdbmstore.c: Likewise.
* src/mmap.c: Likewise.
* src/recover.c: Likewise.
* src/update.c: Likewise.
|
|
* src/gdbmdefs.h (GDBM_SET_ERRNO)
(GDBM_SET_ERRNO2): New macros.
* src/base64.c: Use new macros to set error state.
* src/bucket.c: Likewise.
* src/falloc.c: Likewise.
* src/findkey.c: Likewise.
* src/gdbm_load.c: Likewise.
* src/gdbmdelete.c: Likewise.
* src/gdbmdump.c: Likewise.
* src/gdbmexp.c: Likewise.
* src/gdbmfetch.c: Likewise.
* src/gdbmimp.c: Likewise.
* src/gdbmload.c: Likewise.
* src/gdbmopen.c: Likewise.
* src/gdbmseq.c: Likewise.
* src/gdbmsetopt.c: Likewise.
* src/gdbmstore.c: Likewise.
* src/gdbmtool.c: Fix preprocessor conditional.
|
|
* configure.ac: Fix description wording.
* src/Makefile.am [GDBM_COND_DEBUG_ENABLE]: Don't
define GDBM_DEBUG_ENABLE.
* tests/Makefile.am: Likewise.
* src/debug.c (gdbm_debug_printer)
(gdbm_debug_flags): New globals.
(gdbm_debug_token, gdbm_debug_parse_state)
(gdbm_debug_datum): New functions.
* src/gdbm.h.in [@GDBM_DEBUG_ENABLE@]: Define GDBM_DEBUG_ENABLE.
(gdbm_debug_printer_t): New typedef.
(gdbm_debug_printer, gdbm_debug_flags): New externs.
(GDBM_DEBUG_ERR,GDBM_DEBUG_OPEN)
(GDBM_DEBUG_READ,GDBM_DEBUG_STORE)
(GDBM_DEBUG_LOOKUP,GDBM_DEBUG_ALL): New defines.
(gdbm_debug_token,gdbm_debug_parse_state)
(gdbm_debug_datum): New protos.
* src/gdbmdefs.h (GDBM_DEBUG,GDBM_DEBUG_DATUM): New macros.
* src/findkey.c: Add debugging info.
* src/gdbmfetch.c: Likewise.
* src/gdbmopen.c: Likewise.
* src/gdbmseq.c: Likewise.
* src/gdbmstore.c: Likewise.
* src/gdbmtool.c (open_handler): Allow the use of ~/
(command) <repeat,variadic>: New members.
(run_command): Handle variadic functions.
(run_last_command): New command. In interactive mode,
repeats the last command if it was marked
with repeat=1 (currently, only "next").
New command: "debug".
(all functions): Use terror instead of fprintf(stderr,...);
* src/gdbmtool.h (handler_param) <vararg>: New member.
(run_last_command): New proto.
* src/gram.y: Call run_last_command) on empty input.
* tests/gtload.c: New option: -debug=
|
|
* configure.ac: New option --enable-debug
Print feature summary at the end of the run.
* src/debug.c: New file.
* src/Makefile.am [GDBM_COND_DEBUG_ENABLE]: Build debug.o
Define GDBM_DEBUG_ENABLE.
* src/gdbmdefs.h [GDBM_DEBUG_ENABLE] (_gdbm_debug_hook_install)
(_gdbm_debug_hook_remove,_gdbm_debug_hook_check)
(_gdbm_debug_hook_val): New protos.
(GDBM_DEBUG_HOOK, GDBM_DEBUG_OVERRIDE)
(GDBM_DEBUG_ALLOC): New defines.
* src/gdbm.h.in (GDBM_RCVR_FORCE): New flag.
* src/recover.c (gdbm_recover): Check database before attempting
recovery, unless GDBM_RCVR_FORCE flag is set.
* doc/gdbm.texi: Document GDBM_RCVR_FORCE
* src/gdbmreorg.c (gdbm_reorganize): Use GDBM_RCVR_FORCE.
* src/gdbmtool.c (main): Always allocate file_name.
* src/bucket.c: Put GDBM_DEBUG_OVERRIDE and GDBM_DEBUG_ALLOC
in critical places.
* src/falloc.c: Likewise.
* src/findkey.c: Likewise.
* src/gdbmopen.c: Likewise.
* src/gdbmstore.c: Likewise.
* src/update.c: Likewise.
* tests/Makefile.am [GDBM_COND_DEBUG_ENABLE]: Define GDBM_DEBUG_ENABLE.
* tests/gtload.c: New options -hook, -recover, -verbose,
-backup, -max-failures, -max-failed-keys,
and -max-failed-buckets.
Attempt recovery after errors.
|
|
* src/gdbmconst.h (GDBM_HASH_BITS): New constant.
* src/bucket.c: Use GDBM_HASH_BITS instead of the
hardcoded value.
* src/findkey.c (_gdbm_findkey): Use _gdbm_hash_key.
* src/gdbmtool.c (hash_handler): Use _gdbm_hash_key if the
database is open.
* src/hash.c (_gdbm_bucket_dir, _gdbm_hash_key): New functions.
* src/proto.h (_gdbm_bucket_dir, _gdbm_hash_key): New protos.
* src/systems.h (STATBLKSIZE): Take a struct stat as argument.
* src/gdbmopen.c (STATBLKSIZE): Takes argument now.
|
|
* src/bucket.c (_gdbm_get_bucket)
(_gdbm_split_bucket, _gdbm_write_bucket): Return error code. All callers
updated.
* src/proto.h (_gdbm_get_bucket)
(_gdbm_split_bucket, _gdbm_write_bucket): Update declarations
* src/falloc.c (push_avail_block)
(pop_avail_block): Return error code. All callers
updated.
* src/update.c (_gdbm_fatal): Exit only if the user defined
fatal_err function.
* src/gdbmerrno.c (gdbm_needs_recovery): New function.
* src/gdbm.h.in (gdbm_needs_recovery): New proto.
* compat/dbminit.c: Set gdbm_errno on fatal errors.
* compat/dbmopen.c: Likewise.
* src/findkey.c: Likewise.
* src/gdbm_load.c: Likewise.
* src/gdbmcount.c: Likewise.
* src/gdbmdefs.h: Likewise.
* src/gdbmdelete.c: Likewise.
* src/gdbmdump.c: Likewise.
* src/gdbmexists.c: Likewise.
* src/gdbmexp.c: Likewise.
* src/gdbmfetch.c: Likewise.
* src/gdbmimp.c: Likewise.
* src/gdbmload.c: Likewise.
* src/gdbmopen.c: Likewise.
* src/gdbmreorg.c: Likewise.
* src/gdbmseq.c: Likewise.
* src/gdbmsetopt.c: Likewise.
* src/gdbmstore.c: Likewise.
* src/gdbmsync.c: Likewise.
* src/mmap.c: Likewise.
|
|
Last error code is stored in the database file structure as
well as in the global gdbm_errno. Special functions are provided
for retrieving and clearing the last error state.
* src/gdbmdefs.h (gdbm_file_info): New member: last_error
* src/gdbm.h.in (gdbm_last_errno, gdbm_set_errno)
(gdbm_clear_error): New protos.
* src/gdbmerrno.c (gdbm_last_errno, gdbm_set_errno)
(gdbm_clear_error): New functions
* NEWS: Update.
* compat/dbminit.c: Use gdbm_set_errno to set error state.
* compat/dbmopen.c: Likewise.
* src/bucket.c: Likewise.
* src/findkey.c: Likewise.
* src/gdbm_load.c: Likewise.
* src/gdbmcount.c: Likewise.
* src/gdbmdelete.c: Likewise.
* src/gdbmdump.c: Likewise.
* src/gdbmexists.c: Likewise.
* src/gdbmexp.c: Likewise.
* src/gdbmfetch.c: Likewise.
* src/gdbmimp.c: Likewise.
* src/gdbmload.c: Likewise.
* src/gdbmopen.c: Likewise.
* src/gdbmreorg.c: Likewise.
* src/gdbmseq.c: Likewise.
* src/gdbmsetopt.c: Likewise.
* src/gdbmstore.c: Likewise.
* src/gdbmsync.c: Likewise.
* src/mmap.c: Likewise.
|
|
* doc/gdbm.texi: Document gdbm_exists in detail.
* src/findkey.c (_gdbm_findkey): return parameters can be
NULL.
* src/gdbmexists.c (gdbm_exists): Remove unnecessary
local variables. Reset gdbm_errno to 0 if the key was
not found.
* src/gdbmdelete.c (gdbm_delete): Remove unnecessary
local variables.
* src/gdbmfetch.c: Likewise.
* src/gdbmseq.c: Likewise.
* src/gdbmstore.c: Likewise.
|
|
* src/gdbmfetch.c: Hanlde out of memory error.
* src/findkey.c: Set gdbm_errno to GDBM_ITEM_NOT_FOUND
if nothing was found.
* src/gdbmdelete.c: Don't set gdbm_errno after
_gdbm_findkey returns -1. It's been done already.
* src/gdbmexp.c (gdbm_export_to_file): Return -1 if
gdbm_fetch fails.
* src/gdbmseq.c (get_next_key): Set gdbm_errno to
GDBM_ITEM_NOT_FOUND if there's no next key.
Don't call _gdbm_fatal on out of memory condition.
(gdbm_nextkey): Set gdbm_errno to
GDBM_ITEM_NOT_FOUND if there's no next key.
* src/gdbmtool.c (fetch_handler)
(firstkey_handler,nextkey_handler): Check gdbm_errno.
* src/gdbmstore.c: Handle error return from _gdbm_findkey.
* tests/gtdump.c: Likewise.
* tests/gtfetch.c: Likewise.
* doc/gdbm.texi: Document changes.
* doc/gdbm.3: Likewise.
* configure.ac: Version 1.12.90
* NEWS: Update.
* .gitignore: Update.
|
|
* doc/Config: Rewrite.
* doc/Makefile.am: Use Makeinfo 5 instead of texi2htm
* doc/gendocs_template: Ps is not built
* imprimatur: Upgrade.
|
|
|
|
|
|
Tolerate I/O operations returning less bytes than expected. Retry I/O
if possible.
* src/fullio.c: New file.
* src/Makefile.am (libgdbm_la_SOURCES): Add fullio.c
* src/proto.h (_gdbm_full_read)
(_gdbm_full_write): New protos.
* src/gdbmerrno.c (gdbm_errlist): Add entry for GDBM_FILE_EOF.
* src/bucket.c: Use _gdbm_full_{read|write}.
* src/falloc.c: Likewise.
* src/findkey.c: Likewise.
* src/gdbmopen.c: Likewise.
* src/gdbmstore.c: Likewise.
* src/testgdbm.c: Likewise.
* src/update.c: Likewise.
|
|
* Makefile.am (SUBDIRS): Add po.
(EXTRA_DIST): Add build-aux/config.rpath.
* configure.ac (AC_CONFIG_FILES): Add po/Makefile.in.
* bootstrap: New file.
* src/Makefile.am (AM_CPPFLAGS): Define LOCALEDIR
(noinst_HEADERS): Add gettext.h
(LIBADD): New variable.
* src/gettext.h: New file.
* po/.cvsignore: New file.
* po/Makevars: New file.
* po/POTFILES.in: New file.
* src/gdbmdefs.h: Define DEFAULT_TEXT_DOMAIN, _, N_
Include gettext.h
* src/bucket.c: Add NLS markers.
* src/falloc.c: Likewise.
* src/findkey.c: Likewise.
* src/gdbmerrno.c: Likewise.
* src/gdbmfetch.c: Likewise.
* src/gdbmseq.c: Likewise.
* src/gdbmstore.c: Likewise.
* src/update.c: Likewise.
* src/testgdbm.c: Add NLS markers.
(main): Initialize I18N.
|
|
|
|
|
|
|
