Age | Commit message (Collapse) | Author | Files |
|
|
|
The implementation of _gdbm_cache_flush becomes prohibitively
inefficient during extensive updates of large databases. The
bug was reported at https://github.com/Perl/perl5/issues/19306.
To fix it, make sure that all changed cache entries are placed at
the head of the cache_mru list, forming a contiguous sequence.
This way a potentially long iteration over all cache entries can be
cut off at the first entry with ca_changed == FALSE.
This commit also gets rid of several superfluous fields in
struct gdbm_file_info:
- cache_entry
Not needed, because the most recently used cache entry
(cache_mru) is always the current one.
- bucket_changed
dbf->cache_mru->ca_changed reflects the status of the current
bucket.
- second_changed
Not needed because _gdbm_cache_flush, which flushes all changed
buckets, is now invoked unconditionally by _gdbm_end_update (and
also whenever dbf->cache_mru changes).
* src/gdbmdefs.h (struct gdbm_file_info): Remove cache_entry. The
current cache entry is cache_mru.
Remove bucket_changed, and second_changed.
All uses changed.
* src/proto.h (_gdbm_current_bucket_changed): New inline function.
* src/bucket.c (_gdbm_cache_flush): Assume all changed elements form
a contiguous sequence beginning with dbf->cache_mru.
(set_cache_entry): Remove. All callers changed.
(lru_link_elem,lru_unlink_elem): Update dbf->bucket as necessary.
(cache_lookup): If the obtained bucket is not changed and is going
to become current, flush all changed cache elements.
* src/update.c (_gdbm_end_update): Call _gdbm_cache_flush unconditionally.
* src/findkey.c: Use dbf->cache_mru instead of the removed dbf->cache_entry.
* src/gdbmseq.c: Likewise.
* tools/gdbmshell.c (_gdbm_print_bucket_cache): Likewise.
* src/falloc.c: Use _gdbm_current_bucket_changed to mark the current
bucket as changed.
* src/gdbmstore.c: Likewise.
* src/gdbmdelete.c: Likewise. Use _gdbm_current_bucket_changed.
* tests/gtcacheopt.c: Fix typo.
* tests/gtload.c: New option: -cachesize
|
|
|
|
|
|
This is a very long-standing bug that caused a loss of an avail
entry if the original avail table had odd number of entries.
* src/falloc.c (push_avail_block): Fix computation of dbf->avail->count.
|
|
* src/gdbmconst.h (GDBM_NUMSYNC_MAGIC32)
(GDBM_NUMSYNC_MAGIC64)
(GDBM_NUMSYNC_MAGIC32_SWAP)
(GDBM_NUMSYNC_MAGIC64_SWAP): New constants.
* src/gdbmdefs.h (gdbm_file_header): Move avail to a union in
anticipation of further changes.
(gdbm_file_info) <avail, avail_size>: New members.
(GDBM_HEADER_AVAIL_SIZE): Redefine.
* src/gdbmopen.c (GDBM_NUMSYNC_MAGIC): New define.
(validate_header_std)
(validate_header_numsync): New functions.
(validate_header): Use one of these depending on the magic number.
(gdbm_fd_open): Initialize new members of gdbm_file_info.
* src/avail.c: Use the GDBM_FILE avail pointer.
* src/falloc.c: Likewise.
* src/gdbmtool.c: Likewise.
|
|
This fixes remaining failures in http://puszcza.gnu.org.ua/bugs/?501.
* src/falloc.c (_gdbm_avail_block_read): New function.
(pop_avail_block): Use _gdbm_avail_block_read.
* src/gdbmdefs.h: Fix typo in the comment.
* src/gdbmopen.c (gdbm_avail_block_validate): Take size of the avail
block in bytes as the third argument. All uses changed.
(PARTIAL_HEADER_SIZE)
(HEADER_AVAIL_SIZE): New macros.
(_gdbm_validate_header): Validate the avail block as well.
(validate_header): Don't validate the avail block fields here.
(gdbm_fd_open): Read partial header up to the avail field, then read
entire avail block using _gdbm_avail_block_read.
* src/gdbmtool.c (_gdbm_print_avail_list): Fix avail block calculation
(one extra entry was assumed).
Use _gdbm_avail_block_read to obtain and validate block.
* src/proto.h (_gdbm_avail_block_read): New proto.
(gdbm_avail_block_validate): Change signature.
|
|
|
|
|
|
|
|
* compat/dbmopen.c (ndbm_open_dir_file0): Ignore ENOENT.
* src/falloc.c (push_avail_block): Free temporary storage no matter
what return status.
* src/gdbm.h.in (GDBM_FILE_TRUNCATE_ERROR): New error code.
* src/gdbmdump.c (_gdbm_dump_ascii): Initialize rc.
* src/gdbmerrno.c: Handle new error.code
* src/gdbmload.c (gdbm_load_bdb_dump): Initialize rc
* src/gdbmopen.c (_gdbm_ftruncate): New function.
(gdbm_fd_open): Use _gdbm_ftruncate. Check its return.
* src/gdbmseq.c (gdbm_firstkey): Initialize dsize
* src/gdbmtool.c (command_generator): Check if cmd is NULL.
(shouldn't happen, but anyways).
* src/mmap.c (_gdbm_mapped_lseek): Check for vailidity of the 'whence'
parameter.
* src/systems.h (TRUNCATE): Remove macro.
* src/util.c (vgetyn): Remove unnecessary assignment.
|
|
* src/falloc.c (avail_lookup): Remove the start parameter.
(avail_move): Take pointer to the count of entries in av_table.
Update the pointed to value after performing the move.
All uses changed.
(_gdbm_put_av_elem): Rewrite the "can_merge" loop.
|
|
Use binary search for look-ups.
* src/falloc.c (avail_lookup, avail_move): New functions.
(get_elem): Use avail_lookup to search.
(_gdbm_put_av_elem): Use avail_lookup and avail_move to handle the
avail table. In coalesce mode, store the index of the greater than
or equal entry to avoid extra lookup in case no suitable entry is
found.
|
|
* src/bucket.c (_gdbm_split_bucket): Take into account coalesce_blocks
* src/falloc.c (_gdbm_put_av_elem): Change to void return. All uses changed.
* src/proto.h: Likewise.
|
|
* src/falloc.c (_gdbm_put_av_elem): Float the merged entry to its
proper position to preserve the sorting order.
* THANKS: Thanks Adam Sampson
|
|
* configure.ac: Version 1.5
* NEWS: Update.
* src/Makefile.am (VI_CURRENT): Increment to 6
* src/bucket.c (_gdbm_get_bucket): Fix the upper limit for
bucket_bits.
* src/falloc.c (push_avail_block): Fill the allocated block
with 0s.
|
|
Rename: __read to gdbm_file_read
__write to gdbm_file_write
__lseek to gdbm_file_seek
__fsync to gdbm_file_sync
|
|
The hooks were introduced as a temporary tool in de7834e9. They did
their job and are not necessary any more.
|
|
Verify that avail_table is sorted by size and that each element's
size falls within allowed range.
* src/bucket.c (gdbm_get_bucket): Fix bucket validation. Validate
bucket_avail.
(_gdbm_split_bucket): Check return from _gdbm_free.
* src/falloc.c (adjust_bucket_avail,_gdbm_free): Return error code.
All uses updated.
(pop_avail_block): Fix eventual memory leak. Use gdbm_avail_block_validate.
* src/gdbmdefs.h (gdbm_avail_table_valid_p): Change signature.
* src/gdbmopen.c (gdbm_avail_table_valid_p): Traverse the array verifying
address and size of each element.
(gdbm_avail_block_validate)
(gdbm_bucket_avail_table_validate): New functions.
(validate_header): Remove call to gdbm_avail_block_valid_p. Avail_block
is validated later, after it's been loaded.
Bail out if header->next_block does not equal the file size.
(gdbm_fd_open): Validate avail_block.
* src/gdbmstore.c (_gdbm_store): Check return from _gdbm_free. Avoid
endless loop in case of inconsistent h_table.
* src/gdbmtool.c (_gdbm_print_avail_list): Use gdbm_avail_block_validate.
* src/proto.h: Update.
* tests/gtload.c: Improve error diagnostics.
|
|
* src/gdbmdefs.h (gdbm_avail_block_valid_p): Minimal capacity
of an avail_block is 1 (one entry is stored in the block itself).
* src/gdbmopen.c (gdbm_avail_table_valid_p): New function.
(validate_header): Fix verification of dir_size and dir_bits.
Verify avail.size.
* src/falloc.c (pop_avail_block): Use gdbm_avail_table_valid_p
* src/gdbmtool.c (_gdbm_print_avail_list): Likewise.
|
|
|
|
* src/gdbmdefs.h (gdbm_avail_block_valid_p): New function.
* src/gdbm.h.in (GDBM_BAD_AVAIL): New error code.
* src/gdbmerrno.c: Support new error code.
* src/falloc.c (pop_avail_block): Validate the avail_block
* src/gdbmopen.c (validate_header): Validate the avail_block.
* src/gdbmtool.c (_gdbm_avail_list_size)
(_gdbm_print_avail_list): Validate the avail_block.
|
|
|
|
|
|
Use the GDBM_SET_ERRNO and GDBM_SET_ERRNO2 macros to make
sure the error gets reported in debug output.
* src/fullio.c (_gdbm_full_read)
(_gdbm_full_write): Return -1 and set gdbm_errno
on error.
* src/bucket.c: Use GDBM_SET_ERRNO(2?) or
GDBM_DEBUG where necessary.
* src/falloc.c: Likewise.
* src/findkey.c: Likewise.
* src/gdbmdefs.h: Likewise.
* src/gdbmopen.c: Likewise.
* src/gdbmstore.c: Likewise.
* src/mmap.c: Likewise.
* src/recover.c: Likewise.
* src/update.c: Likewise.
|
|
* src/gdbmdefs.h (GDBM_SET_ERRNO)
(GDBM_SET_ERRNO2): New macros.
* src/base64.c: Use new macros to set error state.
* src/bucket.c: Likewise.
* src/falloc.c: Likewise.
* src/findkey.c: Likewise.
* src/gdbm_load.c: Likewise.
* src/gdbmdelete.c: Likewise.
* src/gdbmdump.c: Likewise.
* src/gdbmexp.c: Likewise.
* src/gdbmfetch.c: Likewise.
* src/gdbmimp.c: Likewise.
* src/gdbmload.c: Likewise.
* src/gdbmopen.c: Likewise.
* src/gdbmseq.c: Likewise.
* src/gdbmsetopt.c: Likewise.
* src/gdbmstore.c: Likewise.
* src/gdbmtool.c: Fix preprocessor conditional.
|
|
* configure.ac: New option --enable-debug
Print feature summary at the end of the run.
* src/debug.c: New file.
* src/Makefile.am [GDBM_COND_DEBUG_ENABLE]: Build debug.o
Define GDBM_DEBUG_ENABLE.
* src/gdbmdefs.h [GDBM_DEBUG_ENABLE] (_gdbm_debug_hook_install)
(_gdbm_debug_hook_remove,_gdbm_debug_hook_check)
(_gdbm_debug_hook_val): New protos.
(GDBM_DEBUG_HOOK, GDBM_DEBUG_OVERRIDE)
(GDBM_DEBUG_ALLOC): New defines.
* src/gdbm.h.in (GDBM_RCVR_FORCE): New flag.
* src/recover.c (gdbm_recover): Check database before attempting
recovery, unless GDBM_RCVR_FORCE flag is set.
* doc/gdbm.texi: Document GDBM_RCVR_FORCE
* src/gdbmreorg.c (gdbm_reorganize): Use GDBM_RCVR_FORCE.
* src/gdbmtool.c (main): Always allocate file_name.
* src/bucket.c: Put GDBM_DEBUG_OVERRIDE and GDBM_DEBUG_ALLOC
in critical places.
* src/falloc.c: Likewise.
* src/findkey.c: Likewise.
* src/gdbmopen.c: Likewise.
* src/gdbmstore.c: Likewise.
* src/update.c: Likewise.
* tests/Makefile.am [GDBM_COND_DEBUG_ENABLE]: Define GDBM_DEBUG_ENABLE.
* tests/gtload.c: New options -hook, -recover, -verbose,
-backup, -max-failures, -max-failed-keys,
and -max-failed-buckets.
Attempt recovery after errors.
|
|
* src/bucket.c (_gdbm_get_bucket)
(_gdbm_split_bucket, _gdbm_write_bucket): Return error code. All callers
updated.
* src/proto.h (_gdbm_get_bucket)
(_gdbm_split_bucket, _gdbm_write_bucket): Update declarations
* src/falloc.c (push_avail_block)
(pop_avail_block): Return error code. All callers
updated.
* src/update.c (_gdbm_fatal): Exit only if the user defined
fatal_err function.
* src/gdbmerrno.c (gdbm_needs_recovery): New function.
* src/gdbm.h.in (gdbm_needs_recovery): New proto.
* compat/dbminit.c: Set gdbm_errno on fatal errors.
* compat/dbmopen.c: Likewise.
* src/findkey.c: Likewise.
* src/gdbm_load.c: Likewise.
* src/gdbmcount.c: Likewise.
* src/gdbmdefs.h: Likewise.
* src/gdbmdelete.c: Likewise.
* src/gdbmdump.c: Likewise.
* src/gdbmexists.c: Likewise.
* src/gdbmexp.c: Likewise.
* src/gdbmfetch.c: Likewise.
* src/gdbmimp.c: Likewise.
* src/gdbmload.c: Likewise.
* src/gdbmopen.c: Likewise.
* src/gdbmreorg.c: Likewise.
* src/gdbmseq.c: Likewise.
* src/gdbmsetopt.c: Likewise.
* src/gdbmstore.c: Likewise.
* src/gdbmsync.c: Likewise.
* src/mmap.c: Likewise.
|
|
|
|
|
|
Tolerate I/O operations returning less bytes than expected. Retry I/O
if possible.
* src/fullio.c: New file.
* src/Makefile.am (libgdbm_la_SOURCES): Add fullio.c
* src/proto.h (_gdbm_full_read)
(_gdbm_full_write): New protos.
* src/gdbmerrno.c (gdbm_errlist): Add entry for GDBM_FILE_EOF.
* src/bucket.c: Use _gdbm_full_{read|write}.
* src/falloc.c: Likewise.
* src/findkey.c: Likewise.
* src/gdbmopen.c: Likewise.
* src/gdbmstore.c: Likewise.
* src/testgdbm.c: Likewise.
* src/update.c: Likewise.
|
|
* Makefile.am (SUBDIRS): Add po.
(EXTRA_DIST): Add build-aux/config.rpath.
* configure.ac (AC_CONFIG_FILES): Add po/Makefile.in.
* bootstrap: New file.
* src/Makefile.am (AM_CPPFLAGS): Define LOCALEDIR
(noinst_HEADERS): Add gettext.h
(LIBADD): New variable.
* src/gettext.h: New file.
* po/.cvsignore: New file.
* po/Makevars: New file.
* po/POTFILES.in: New file.
* src/gdbmdefs.h: Define DEFAULT_TEXT_DOMAIN, _, N_
Include gettext.h
* src/bucket.c: Add NLS markers.
* src/falloc.c: Likewise.
* src/findkey.c: Likewise.
* src/gdbmerrno.c: Likewise.
* src/gdbmfetch.c: Likewise.
* src/gdbmseq.c: Likewise.
* src/gdbmstore.c: Likewise.
* src/update.c: Likewise.
* src/testgdbm.c: Add NLS markers.
(main): Initialize I18N.
|
|
|
|
|
|
|