2008-05-10 Ulrich Drepper <drepper@redhat.com>
* sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_rdlock.S: Access
__pshared correctly.
* sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_timedrdlock.S:
Likewise.
* sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_timedwrlock.S:
Likewise.
* sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_unlock.S:
Likewise.
* sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_wrlock.S:
Likewise.
Reported by Clemens Kolbitsch <clemens.kol@gmx.at>.
2008-04-28 Hiroki Kaminaga <kaminaga@sm.sony.co.jp>
[BZ #6740]
* sysdeps/powerpc/tcb-offsets.sym (PRIVATE_FUTEX_OFFSET): Guard symbol
definition with #ifndef __ASSUME_PRIVATE_FUTEX.
2007-10-17 Jakub Jelinek <jakub@redhat.com>
* sysdeps/unix/sysv/linux/i386/i486/sem_post.S (__old_sem_post): New
routine instead of an alias to __new_sem_post.
2007-10-15 Jakub Jelinek <jakub@redhat.com>
* init.c (__pthread_initialize_minimal): Initialize word to appease
valgrind.
2007-09-02 Ulrich Drepper <drepper@redhat.com>
* sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S
(__pthread_cond_wait): Fix unlocking of internal lock after mutex
unlocking failed.
Patch by Luca Barbieri <luca.barbieri@gmail.com>.
2007-08-14 Jakub Jelinek <jakub@redhat.com>
* sysdeps/unix/sysv/linux/lowlevellock.c: Comment fix.
* sysdeps/unix/sysv/linux/s390/lowlevellock.h (__lll_unlock,
__lll_robust_unlock, __lll_wait_tid): Rewrite as macros instead of
inline functions.
2007-08-13 Jakub Jelinek <jakub@redhat.com>
* sysdeps/unix/sysv/linux/i386/lowlevellock.h (__lll_private_flag):
Fix a pasto.
* sysdeps/unix/sysv/linux/i386/i486/pthread_cond_broadcast.S
(__pthread_cond_broadcast): Pass LLL_PRIVATE to lll_* and or
FUTEX_PRIVATE_FLAG into SYS_futex op if cv is process private.
Don't use FUTEX_CMP_REQUEUE if dep_mutex is not process private.
* sysdeps/unix/sysv/linux/i386/i486/pthread_cond_signal.S
(__pthread_cond_signal): Pass LLL_PRIVATE to lll_* and or
FUTEX_PRIVATE_FLAG into SYS_futex op if cv is process private.
* sysdeps/unix/sysv/linux/i386/i486/pthread_cond_wait.S: Include
kernel-features.h.
(__pthread_cond_wait, __condvar_w_cleanup): Pass LLL_PRIVATE to
lll_* and or FUTEX_PRIVATE_FLAG into SYS_futex op if cv is
process private. Switch DW_CFA_advance_loc1 and some
DW_CFA_advance_loc .eh_frame opcodes to DW_CFA_advance_loc4.
* sysdeps/unix/sysv/linux/i386/i486/pthread_cond_timedwait.S
(__pthread_cond_timedwait, __condvar_tw_cleanup): Pass LLL_PRIVATE to
lll_* and or FUTEX_PRIVATE_FLAG into SYS_futex op if cv is
process private. Switch DW_CFA_advance_loc{1,2} and some
DW_CFA_advance_loc .eh_frame opcodes to DW_CFA_advance_loc4.
* sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_rdlock.S: Use
#ifdef __ASSUME_PRIVATE_FUTEX instead of #if __ASSUME_PRIVATE_FUTEX.
* sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_timedwrlock.S:
Likewise.
* sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_unlock.S: Likewise.
* sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_wrlock.S: Likewise.
* sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_timedrdlock.S:
Likewise.
* sysdeps/unix/sysv/linux/x86_64/pthread_cond_broadcast.S
(__pthread_cond_broadcast): Compare %r8 instead of
dep_mutex-cond_*(%rdi) with $-1.
* sysdeps/unix/sysv/linux/x86_64/pthread_cond_signal.S
(__pthread_cond_signal): Xor FUTEX_WAKE_OP with FUTEX_WAKE instead
of oring.
2007-08-13 Jakub Jelinek <jakub@redhat.com>
* allocatestack.c: Include kernel-features.h.
* pthread_create.c: Likewise.
* pthread_mutex_init.c: Likewise.
* init.c: Likewise.
* pthread_cond_timedwait.c: Likewise.
* sysdeps/unix/sysv/linux/ia64/lowlevellock.h: Likewise.
* sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_wrlock.S: Likewise.
* sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_rdlock.S: Likewise.
* sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_unlock.S: Likewise.
* sysdeps/unix/sysv/linux/i386/i486/pthread_cond_timedwait.S: Likewise.
* sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_timedwrlock.S:
Likewise.
* sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_timedrdlock.S:
Likewise.
* sysdeps/unix/sysv/linux/s390/lowlevellock.h: Likewise.
* sysdeps/unix/sysv/linux/powerpc/lowlevellock.h: Likewise.
2007-08-12 Jakub Jelinek <jakub@redhat.com>
* sysdeps/unix/sysv/linux/s390/bits/pthreadtypes.h
[__WORDSIZE=32] (pthread_rwlock_t): Split __flags element into four
byte elements. One of them is the new __shared element.
[__WORDSIZE=64] (pthread_rwlock_t): Renamed __pad1 element to __shared,
adjust names of other padding elements.
* sysdeps/unix/sysv/linux/ia64/bits/pthreadtypes.h (pthread_rwlock_t):
Renamed __pad1 element to __shared, adjust names of other padding
elements.
* sysdeps/unix/sysv/linux/ia64/lowlevellock.h (__lll_lock): Fix a
typo.
2007-08-12 Ulrich Drepper <drepper@redhat.com>
* sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S: Include
<kernel-features.h>.
* sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S: Likewise.
2007-08-11 Ulrich Drepper <drepper@redhat.com>
* pthreadP.h (PTHREAD_ROBUST_MUTEX_PSHARED): Define.
* pthread_mutex_lock.c: Use it instead of PTHREAD_MUTEX_PSHARED when
dealing with robust mutexes.
* pthread_mutex_timedlock.c: Likewise.
* pthread_mutex_trylock.c: Likewise.
* pthread_mutex_unlock.c: Likewise.
* sysdeps/unix/sysv/linux/pthread_mutex_cond_lock.c: Likewise.
2007-08-06 Jakub Jelinek <jakub@redhat.com>
* pthreadP.h (PTHREAD_MUTEX_PSHARED_BIT): Define.
(PTHREAD_MUTEX_TYPE): Mask __kind with 127.
(PTHREAD_MUTEX_PSHARED): Define.
* pthread_mutex_init.c (__pthread_mutex_init): Set
PTHREAD_MUTEX_PSHARED_BIT for pshared or robust
mutexes.
* pthread_mutex_lock.c (LLL_MUTEX_LOCK): Take mutex as argument
instead of its __data.__lock field, pass PTHREAD_MUTEX_PSHARED
as second argument to lll_lock.
(LLL_MUTEX_TRYLOCK): Take mutex as argument
instead of its __data.__lock field.
(LLL_ROBUST_MUTEX_LOCK): Take mutex as argument instead of its
__data.__lock field, pass PTHREAD_MUTEX_PSHARED as second argument
to lll_robust_lock.
(__pthread_mutex_lock): Update LLL_MUTEX_LOCK, LLL_MUTEX_TRYLOCK,
LLL_ROBUST_MUTEX_LOCK users, use PTHREAD_MUTEX_TYPE (mutex)
instead of mutex->__data.__kind directly, pass
PTHREAD_MUTEX_PSHARED (mutex) to lll_unlock and lll_futex_wait.
* pthread_mutex_trylock.c (__pthread_mutex_trylock): Use
PTHREAD_MUTEX_TYPE (mutex) instead of mutex->__data.__kind
directly, pass PTHREAD_MUTEX_PSHARED (mutex) to lll_unlock.
(pthread_mutex_timedlock): Pass PTHREAD_MUTEX_PSHARED (mutex)
to lll_timedlock, lll_robust_timedlock, lll_unlock and
lll_futex_timed_wait. Use PTHREAD_MUTEX_TYPE (mutex) instead
of mutex->__data.__kind directly.
* pthread_mutex_timedlock.c (pthread_mutex_timedlock): Pass
PTHREAD_MUTEX_PSHARED (mutex) to lll_timedlock,
lll_robust_timedlock, lll_unlock and lll_futex_timed_wait. Use
PTHREAD_MUTEX_TYPE (mutex) instead of mutex->__data.__kind directly.
* pthread_mutex_unlock.c (__pthread_mutex_unlock_usercnt): Pass
PTHREAD_MUTEX_PSHARED (mutex) to lll_unlock, lll_robust_unlock
and lll_futex_wake.
* pthread_mutex_setprioceiling.c (pthread_mutex_setprioceiling): Pass
PTHREAD_MUTEX_PSHARED (mutex) to lll_futex_wait and lll_futex_wake.
Use PTHREAD_MUTEX_TYPE (mutex) instead of mutex->__data.__kind
directly.
* sysdeps/unix/sysv/linux/pthread_mutex_cond_lock.c (LLL_MUTEX_LOCK):
Take mutex as argument instead of its __data.__lock field, pass
PTHREAD_MUTEX_PSHARED as second argument to lll_cond_lock.
(LLL_MUTEX_TRYLOCK): Take mutex as argument instead of its
__data.__lock field.
(LLL_ROBUST_MUTEX_LOCK): Take mutex as argument instead of its
__data.__lock field, pass PTHREAD_MUTEX_PSHARED as second argument
to lll_robust_cond_lock.
* pthread_cond_broadcast.c (__pthread_cond_broadcast): Add pshared
variable, pass it to lll_lock, lll_unlock, lll_futex_requeue and
lll_futex_wake. Don't use lll_futex_requeue if dependent mutex
has PTHREAD_MUTEX_PSHARED_BIT bit set in its __data.__kind.
* pthread_cond_destroy.c (__pthread_cond_destroy): Add pshared
variable, pass it to lll_lock, lll_unlock, lll_futex_wake and
lll_futex_wait.
* pthread_cond_signal.c (__pthread_cond_signal): Add pshared
variable, pass it to lll_lock, lll_unlock, lll_futex_wake_unlock and
lll_futex_wake.
* pthread_cond_timedwait.c (__pthread_cond_wait): Add
pshared variable, pass it to lll_lock, lll_unlock,
lll_futex_timedwait and lll_futex_wake.
* pthread_cond_wait.c (__condvar_cleanup, __pthread_cond_wait): Add
pshared variable, pass it to lll_lock, lll_unlock, lll_futex_wait
and lll_futex_wake.
* sysdeps/unix/sysv/linux/ia64/lowlevellock.h (lll_futex_requeue,
lll_futex_wake_unlock): Likewise.
* sysdeps/unix/sysv/linux/powerpc/lowlevellock.h (lll_futex_requeue):
Likewise.
* sysdeps/unix/sysv/linux/x86_64/lowlevellock.h (lll_futex_requeue):
Likewise.
* sysdeps/unix/sysv/linux/s390/lowlevellock.h (lll_futex_requeue,
lll_futex_wake_unlock): Likewise.
(lll_futex_wake): Fix a typo.
* sysdeps/unix/sysv/linux/pthread-pi-defines.sym (PS_BIT): Add.
* sysdeps/unix/sysv/linux/x86_64/pthread_cond_broadcast.S
(__pthread_cond_broadcast): Pass LLL_PRIVATE to lll_* and or
FUTEX_PRIVATE_FLAG into SYS_futex op if cv is process private.
Don't use FUTEX_CMP_REQUEUE if dep_mutex is not process private.
* sysdeps/unix/sysv/linux/x86_64/pthread_cond_signal.S
(__pthread_cond_signal): Pass LLL_PRIVATE to lll_* and or
FUTEX_PRIVATE_FLAG into SYS_futex op if cv is process private.
* sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S
(__pthread_cond_timedwait): Likewise.
* sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:
(__condvar_cleanup, __pthread_cond_wait): Likewise.
2007-07-31 Anton Blanchard <anton@samba.org>
* sysdeps/unix/sysv/linux/powerpc/sem_post.c (__new_sem_post):
Use __asm __volatile (__lll_acq_instr ::: "memory") instead of
atomic_full_barrier.
2007-07-31 Jakub Jelinek <jakub@redhat.com>
* allocatestack.c (stack_cache_lock): Change type to int.
(get_cached_stack, allocate_stack, __deallocate_stack,
__make_stacks_executable, __find_thread_by_id, __nptl_setxid,
__pthread_init_static_tls, __wait_lookup_done): Add LLL_PRIVATE
as second argument to lll_lock and lll_unlock macros on
stack_cache_lock.
* pthread_create.c (__find_in_stack_list): Likewise.
(start_thread): Similarly with pd->lock. Use lll_robust_dead
macro instead of lll_robust_mutex_dead, pass LLL_SHARED to it
as second argument.
* descr.h (struct pthread): Change lock and setxid_futex field
type to int.
* old_pthread_cond_broadcast.c (__pthread_cond_broadcast_2_0): Use
LLL_LOCK_INITIALIZER instead of LLL_MUTEX_LOCK_INITIALIZER.
* old_pthread_cond_signal.c (__pthread_cond_signal_2_0): Likewise.
* old_pthread_cond_timedwait.c (__pthread_cond_timedwait_2_0):
Likewise.
* old_pthread_cond_wait.c (__pthread_cond_wait_2_0): Likewise.
* pthread_cond_init.c (__pthread_cond_init): Likewise.
* pthreadP.h (__attr_list_lock): Change type to int.
* pthread_attr_init.c (__attr_list_lock): Likewise.
* pthread_barrier_destroy.c (pthread_barrier_destroy): Pass
ibarrier->private ^ FUTEX_PRIVATE_FLAG as second argument to
lll_{,un}lock.
* pthread_barrier_wait.c (pthread_barrier_wait): Likewise and
also for lll_futex_{wake,wait}.
* pthread_barrier_init.c (pthread_barrier_init): Make iattr
a pointer to const.
* pthread_cond_broadcast.c (__pthread_cond_broadcast): Pass
LLL_SHARED as second argument to lll_{,un}lock.
* pthread_cond_destroy.c (__pthread_cond_destroy): Likewise.
* pthread_cond_signal.c (__pthread_cond_singal): Likewise.
* pthread_cond_timedwait.c (__pthread_cond_timedwait): Likewise.
* pthread_cond_wait.c (__condvar_cleanup, __pthread_cond_wait):
Likewise.
* pthread_getattr_np.c (pthread_getattr_np): Add LLL_PRIVATE
as second argument to lll_{,un}lock macros on pd->lock.
* pthread_getschedparam.c (__pthread_getschedparam): Likewise.
* pthread_setschedparam.c (__pthread_setschedparam): Likewise.
* pthread_setschedprio.c (pthread_setschedprio): Likewise.
* tpp.c (__pthread_tpp_change_priority, __pthread_current_priority):
Likewise.
* sysdeps/pthread/createthread.c (do_clone, create_thread):
Likewise.
* pthread_once.c (once_lock): Change type to int.
(__pthread_once): Pass LLL_PRIVATE as second argument to
lll_{,un}lock macros on once_lock.
* pthread_rwlock_rdlock.c (__pthread_rwlock_rdlock): Use
lll_{,un}lock macros instead of lll_mutex_{,un}lock, pass
rwlock->__data.__shared as second argument to them and similarly
for lll_futex_w*.
* pthread_rwlock_timedrdlock.c (pthread_rwlock_timedrdlock):
Likewise.
* pthread_rwlock_timedwrlock.c (pthread_rwlock_timedwrlock):
Likewise.
* pthread_rwlock_tryrdlock.c (__pthread_rwlock_tryrdlock): Likewise.
* pthread_rwlock_trywrlock.c (__pthread_rwlock_trywrlock): Likewise.
* pthread_rwlock_unlock.c (__pthread_rwlock_unlock): Likewise.
* pthread_rwlock_wrlock.c (__pthread_rwlock_wrlock): Likewise.
* sem_close.c (sem_close): Pass LLL_PRIVATE as second argument
to lll_{,un}lock macros on __sem_mappings_lock.
* sem_open.c (check_add_mapping): Likewise.
(__sem_mappings_lock): Change type to int.
* semaphoreP.h (__sem_mappings_lock): Likewise.
* pthread_mutex_lock.c (LLL_MUTEX_LOCK, LLL_MUTEX_TRYLOCK,
LLL_ROBUST_MUTEX_LOCK): Use lll_{,try,robust_}lock macros
instead of lll_*mutex_*, pass LLL_SHARED as last
argument.
(__pthread_mutex_lock): Use lll_unlock instead of lll_mutex_unlock,
pass LLL_SHARED as last argument.
* sysdeps/unix/sysv/linux/pthread_mutex_cond_lock.c (LLL_MUTEX_LOCK,
LLL_MUTEX_TRYLOCK, LLL_ROBUST_MUTEX_LOCK): Use
lll_{cond_,cond_try,robust_cond}lock macros instead of lll_*mutex_*,
pass LLL_SHARED as last argument.
* pthread_mutex_timedlock.c (pthread_mutex_timedlock): Use
lll_{timed,try,robust_timed,un}lock instead of lll_*mutex*, pass
LLL_SHARED as last argument.
* pthread_mutex_trylock.c (__pthread_mutex_trylock): Similarly.
* pthread_mutex_unlock.c (__pthread_mutex_unlock_usercnt):
Similarly.
* sysdeps/pthread/bits/libc-lock.h (__libc_lock_lock,
__libc_lock_lock_recursive, __libc_lock_unlock,
__libc_lock_unlock_recursive): Pass LLL_PRIVATE as second
argument to lll_{,un}lock.
* sysdeps/pthread/bits/stdio-lock.h (_IO_lock_lock,
_IO_lock_unlock): Likewise.
* sysdeps/unix/sysv/linux/fork.c (__libc_fork): Don't use
compound literal.
* sysdeps/unix/sysv/linux/unregister-atfork.c (__unregister_atfork):
Pass LLL_PRIVATE as second argument to lll_{,un}lock macros on
__fork_lock.
* sysdeps/unix/sysv/linux/register-atfork.c (__register_atfork,
free_mem): Likewise.
(__fork_lock): Change type to int.
* sysdeps/unix/sysv/linux/fork.h (__fork_lock): Likewise.
* sysdeps/unix/sysv/linux/sem_post.c (__new_sem_post): Pass
isem->private ^ FUTEX_PRIVATE_FLAG as second argument to
lll_futex_wake.
* sysdeps/unix/sysv/linux/sem_timedwait.c (sem_timedwait): Likewise.
* sysdeps/unix/sysv/linux/sem_wait.c (__new_sem_wait): Likewise.
* sysdeps/unix/sysv/linux/lowlevellock.c (__lll_lock_wait_private):
New function.
(__lll_lock_wait, __lll_timedlock_wait): Add private argument and
pass it through to lll_futex_*wait, only compile in when
IS_IN_libpthread.
* sysdeps/unix/sysv/linux/lowlevelrobustlock.c
(__lll_robust_lock_wait, __lll_robust_timedlock_wait): Add private
argument and pass it through to lll_futex_*wait.
* sysdeps/unix/sysv/linux/ia64/lowlevellock.h: Renamed all
lll_mutex_* resp. lll_robust_mutex_* macros to lll_* resp.
lll_robust_*. Renamed all __lll_mutex_* resp. __lll_robust_mutex_*
inline functions to __lll_* resp. __lll_robust_*.
(LLL_MUTEX_LOCK_INITIALIZER): Remove.
(lll_mutex_dead): Add private argument.
(__lll_lock_wait_private): New prototype.
(__lll_lock_wait, __lll_robust_lock_wait, __lll_lock_timedwait,
__lll_robust_lock_timedwait): Add private argument to prototypes.
(__lll_lock): Add private argument, if it is constant LLL_PRIVATE,
call __lll_lock_wait_private, otherwise pass private to
__lll_lock_wait.
(__lll_robust_lock, __lll_cond_lock, __lll_timedlock,
__lll_robust_timedlock): Add private argument, pass it to
__lll_*wait functions.
(__lll_unlock): Add private argument, if it is constant LLL_PRIVATE,
call __lll_unlock_wake_private, otherwise pass private to
__lll_unlock_wake.
(__lll_robust_unlock): Add private argument, pass it to
__lll_robust_unlock_wake.
(lll_lock, lll_robust_lock, lll_cond_lock, lll_timedlock,
lll_robust_timedlock, lll_unlock, lll_robust_unlock): Add private
argument, pass it through to __lll_* inline function.
(__lll_mutex_unlock_force, lll_mutex_unlock_force): Remove.
(lll_lock_t): Remove.
(__lll_cond_wait, __lll_cond_timedwait, __lll_cond_wake,
__lll_cond_broadcast, lll_cond_wait, lll_cond_timedwait,
lll_cond_wake, lll_cond_broadcast): Remove.
* sysdeps/unix/sysv/linux/powerpc/lowlevellock.h: Likewise.
* sysdeps/unix/sysv/linux/s390/lowlevellock.h: Likewise.
* sysdeps/unix/sysv/linux/i386/lowlevellock.h: Allow including
the header from assembler. Renamed all lll_mutex_* resp.
lll_robust_mutex_* macros to lll_* resp. lll_robust_*.
(LOCK, FUTEX_CMP_REQUEUE, FUTEX_WAKE_OP,
FUTEX_OP_CLEAR_WAKE_IF_GT_ONE): Define.
(LLL_MUTEX_LOCK_INITIALIZER, LLL_MUTEX_LOCK_INITIALIZER_LOCKED,
LLL_MUTEX_LOCK_INITIALIZER_WAITERS): Remove.
(__lll_mutex_lock_wait, __lll_mutex_timedlock_wait,
__lll_mutex_unlock_wake, __lll_lock_wait, __lll_unlock_wake):
Remove prototype.
(__lll_trylock_asm, __lll_lock_asm_start, __lll_unlock_asm): Define.
(lll_robust_trylock, lll_cond_trylock): Use LLL_LOCK_INITIALIZER*
rather than LLL_MUTEX_LOCK_INITIALIZER* macros.
(lll_trylock): Likewise, use __lll_trylock_asm, pass
MULTIPLE_THREADS_OFFSET as another asm operand.
(lll_lock): Add private argument, use __lll_lock_asm_start, pass
MULTIPLE_THREADS_OFFSET as last asm operand, call
__lll_lock_wait_private if private is constant LLL_PRIVATE,
otherwise pass private as another argument to __lll_lock_wait.
(lll_robust_lock, lll_cond_lock, lll_robust_cond_lock,
lll_timedlock, lll_robust_timedlock): Add private argument, pass
private as another argument to __lll_*lock_wait call.
(lll_unlock): Add private argument, use __lll_unlock_asm, pass
MULTIPLE_THREADS_OFFSET as another asm operand, call
__lll_unlock_wake_private if private is constant LLL_PRIVATE,
otherwise pass private as another argument to __lll_unlock_wake.
(lll_robust_unlock): Add private argument, pass private as another
argument to __lll_unlock_wake.
(lll_robust_dead): Add private argument, use __lll_private_flag
macro.
(lll_islocked): Use LLL_LOCK_INITIALIZER instead of
LLL_MUTEX_LOCK_INITIALIZER.
(lll_lock_t): Remove.
(LLL_LOCK_INITIALIZER_WAITERS): Define.
(__lll_cond_wait, __lll_cond_timedwait, __lll_cond_wake,
__lll_cond_broadcast, lll_cond_wait, lll_cond_timedwait,
lll_cond_wake, lll_cond_broadcast): Remove.
* sysdeps/unix/sysv/linux/x86_64/lowlevellock.h: Likewise.
* sysdeps/unix/sysv/linux/i386/i486/libc-lowlevellock.S: Revert
2007-05-2{3,9} changes.
* sysdeps/unix/sysv/linux/i386/i486/lowlevellock.S: Include
kernel-features.h and lowlevellock.h.
(LOAD_PRIVATE_FUTEX_WAIT): Define.
(LOAD_FUTEX_WAIT): Rewritten.
(LOCK, SYS_gettimeofday, SYS_futex, FUTEX_WAIT, FUTEX_WAKE): Don't
define.
(__lll_lock_wait_private, __lll_unlock_wake_private): New functions.
(__lll_mutex_lock_wait): Rename to ...
(__lll_lock_wait): ... this. Take futex addr from %edx instead of
%ecx, %ecx is now private argument. Don't compile in for libc.so.
(__lll_mutex_timedlock_wait): Rename to ...
(__lll_timedlock_wait): ... this. Use __NR_gettimeofday. %esi
contains private argument. Don't compile in for libc.so.
(__lll_mutex_unlock_wake): Rename to ...
(__lll_unlock_wake): ... this. %ecx contains private argument.
Don't compile in for libc.so.
(__lll_timedwait_tid): Use __NR_gettimeofday.
* sysdeps/unix/sysv/linux/i386/i486/lowlevelrobustlock.S: Include
kernel-features.h and lowlevellock.h.
(LOAD_FUTEX_WAIT): Define.
(LOCK, SYS_gettimeofday, SYS_futex, FUTEX_WAIT, FUTEX_WAKE): Don't
define.
(__lll_robust_mutex_lock_wait): Rename to ...
(__lll_robust_lock_wait): ... this. Futex addr is now in %edx
argument, %ecx argument contains private. Use LOAD_FUTEX_WAIT
macro.
(__lll_robust_mutex_timedlock_wait): Rename to ...
(__lll_robust_timedlock_wait): ... this. Use __NR_gettimeofday.
%esi argument contains private, use LOAD_FUTEX_WAIT macro.
* sysdeps/unix/sysv/linux/i386/i486/pthread_barrier_wait.S: Include
lowlevellock.h.
(SYS_futex, FUTEX_WAIT, FUTEX_WAKE, LOCK): Don't define.
(pthread_barrier_wait): Rename __lll_mutex_* to __lll_*, pass
PRIVATE(%ebx) ^ LLL_SHARED as private argument in %ecx to
__lll_lock_wait and __lll_unlock_wake, pass MUTEX(%ebx) address
to __lll_lock_wait in %edx.
* sysdeps/unix/sysv/linux/i386/i486/pthread_cond_broadcast.S:
Include lowlevellock.h and pthread-errnos.h.
(SYS_futex, FUTEX_WAIT, FUTEX_WAKE, FUTEX_REQUEUE,
FUTEX_CMP_REQUEUE, EINVAL, LOCK): Don't define.
(__pthread_cond_broadcast): Rename __lll_mutex_* to __lll_*, pass
cond_lock address in %edx rather than %ecx to __lll_lock_wait,
pass LLL_SHARED in %ecx to both __lll_lock_wait and
__lll_unlock_wake.
* sysdeps/unix/sysv/linux/i386/i486/pthread_cond_signal.S:
Include lowlevellock.h and pthread-errnos.h.
(SYS_futex, FUTEX_WAIT, FUTEX_WAKE, FUTEX_WAKE_OP,
FUTEX_OP_CLEAR_WAKE_IF_GT_ONE, EINVAL, LOCK): Don't define.
(__pthread_cond_signal): Rename __lll_mutex_* to __lll_*, pass
cond_lock address in %edx rather than %ecx to __lll_lock_wait,
pass LLL_SHARED in %ecx to both __lll_lock_wait and
__lll_unlock_wake.
* sysdeps/unix/sysv/linux/i386/i486/pthread_cond_timedwait.S:
Include lowlevellock.h.
(SYS_futex, SYS_gettimeofday, FUTEX_WAIT, FUTEX_WAKE, LOCK):
Don't define.
(__pthread_cond_timedwait): Rename __lll_mutex_* to __lll_*, pass
cond_lock address in %edx rather than %ecx to __lll_lock_wait,
pass LLL_SHARED in %ecx to both __lll_lock_wait and
__lll_unlock_wake. Use __NR_gettimeofday.
* sysdeps/unix/sysv/linux/i386/i486/pthread_cond_wait.S:
Include lowlevellock.h.
(SYS_futex, FUTEX_WAIT, FUTEX_WAKE, LOCK): Don't define.
(__pthread_cond_wait, __condvar_w_cleanup): Rename __lll_mutex_*
to __lll_*, pass cond_lock address in %edx rather than %ecx to
__lll_lock_wait, pass LLL_SHARED in %ecx to both __lll_lock_wait
and __lll_unlock_wake.
* sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_rdlock.S:
Include lowlevellock.h.
(SYS_futex, FUTEX_WAIT, FUTEX_WAKE, LOCK): Don't define.
(__pthread_rwlock_rdlock): Rename __lll_mutex_* to __lll_*, pass
MUTEX(%ebx) address in %edx rather than %ecx to
__lll_lock_wait, pass PSHARED(%ebx) in %ecx to both __lll_lock_wait
and __lll_unlock_wake. Move return value from %ecx to %edx
register.
* sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_timedrdlock.S:
Include lowlevellock.h.
(SYS_futex, SYS_gettimeofday, FUTEX_WAIT, FUTEX_WAKE, LOCK):
Don't define.
(__pthread_rwlock_wrlock): Rename __lll_mutex_* to __lll_*, pass
MUTEX(%ebp) address in %edx rather than %ecx to
__lll_lock_wait, pass PSHARED(%ebp) in %ecx to both __lll_lock_wait
and __lll_unlock_wake. Move return value from %ecx to %edx
register. Use __NR_gettimeofday.
* sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_timedwrlock.S:
Include lowlevellock.h.
(SYS_futex, SYS_gettimeofday, FUTEX_WAIT, FUTEX_WAKE, LOCK):
Don't define.
(__pthread_rwlock_wrlock): Rename __lll_mutex_* to __lll_*, pass
MUTEX(%ebp) address in %edx rather than %ecx to
__lll_lock_wait, pass PSHARED(%ebp) in %ecx to both __lll_lock_wait
and __lll_unlock_wake. Move return value from %ecx to %edx
register. Use __NR_gettimeofday.
* sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_unlock.S:
Include lowlevellock.h.
(SYS_futex, FUTEX_WAIT, FUTEX_WAKE, LOCK): Don't define.
(__pthread_rwlock_unlock): Rename __lll_mutex_* to __lll_*, pass
MUTEX(%edi) address in %edx rather than %ecx to
__lll_lock_wait, pass PSHARED(%edi) in %ecx to both __lll_lock_wait
and __lll_unlock_wake.
* sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_wrlock.S:
Include lowlevellock.h.
(SYS_futex, FUTEX_WAIT, FUTEX_WAKE, LOCK): Don't define.
(__pthread_rwlock_wrlock): Rename __lll_mutex_* to __lll_*, pass
MUTEX(%ebx) address in %edx rather than %ecx to
__lll_lock_wait, pass PSHARED(%ebx) in %ecx to both __lll_lock_wait
and __lll_unlock_wake. Move return value from %ecx to %edx
register.
* sysdeps/unix/sysv/linux/i386/pthread_once.S: Include
lowlevellock.h.
(LOCK, SYS_futex, FUTEX_WAIT, FUTEX_WAKE, FUTEX_PRIVATE_FLAG): Don't
define.
* sysdeps/unix/sysv/linux/i386/i486/sem_post.S: Include lowlevellock.h.
(LOCK, SYS_futex, FUTEX_WAKE): Don't define.
* sysdeps/unix/sysv/linux/i386/i486/sem_timedwait.S: Include
lowlevellock.h.
(LOCK, SYS_futex, SYS_gettimeofday, FUTEX_WAIT): Don't define.
(sem_timedwait): Use __NR_gettimeofday.
* sysdeps/unix/sysv/linux/i386/i486/sem_trywait.S: Include
lowlevellock.h.
(LOCK): Don't define.
* sysdeps/unix/sysv/linux/i386/i486/sem_wait.S: Include
lowlevellock.h.
(LOCK, SYS_futex, FUTEX_WAIT): Don't define.
* sysdeps/unix/sysv/linux/powerpc/sem_post.c: Wake only when there
are waiters.
* sysdeps/unix/sysv/linux/x86_64/libc-lowlevellock.S: Revert
2007-05-2{3,9} changes.
* sysdeps/unix/sysv/linux/x86_64/lowlevellock.S: Include
kernel-features.h and lowlevellock.h.
(LOAD_PRIVATE_FUTEX_WAIT): Define.
(LOAD_FUTEX_WAIT): Rewritten.
(LOCK, SYS_futex, FUTEX_WAIT, FUTEX_WAKE): Don't define.
(__lll_lock_wait_private, __lll_unlock_wake_private): New functions.
(__lll_mutex_lock_wait): Rename to ...
(__lll_lock_wait): ... this. %esi is now private argument.
Don't compile in for libc.so.
(__lll_mutex_timedlock_wait): Rename to ...
(__lll_timedlock_wait): ... this. %esi contains private argument.
Don't compile in for libc.so.
(__lll_mutex_unlock_wake): Rename to ...
(__lll_unlock_wake): ... this. %esi contains private argument.
Don't compile in for libc.so.
* sysdeps/unix/sysv/linux/x86_64/lowlevelrobustlock.S: Include
kernel-features.h and lowlevellock.h.
(LOAD_FUTEX_WAIT): Define.
(LOCK, SYS_futex, FUTEX_WAIT, FUTEX_WAKE): Don't define.
(__lll_robust_mutex_lock_wait): Rename to ...
(__lll_robust_lock_wait): ... this. %esi argument contains private.
Use LOAD_FUTEX_WAIT macro.
(__lll_robust_mutex_timedlock_wait): Rename to ...
(__lll_robust_timedlock_wait): ... this. %esi argument contains
private, use LOAD_FUTEX_WAIT macro.
* sysdeps/unix/sysv/linux/x86_64/pthread_barrier_wait.S: Include
lowlevellock.h.
(SYS_futex, FUTEX_WAIT, FUTEX_WAKE, LOCK): Don't define.
(pthread_barrier_wait): Rename __lll_mutex_* to __lll_*, pass
PRIVATE(%rdi) ^ LLL_SHARED as private argument in %esi to
__lll_lock_wait and __lll_unlock_wake.
* sysdeps/unix/sysv/linux/x86_64/pthread_cond_broadcast.S:
Include lowlevellock.h and pthread-errnos.h.
(SYS_futex, FUTEX_WAIT, FUTEX_WAKE, FUTEX_REQUEUE,
FUTEX_CMP_REQUEUE, EINVAL, LOCK): Don't define.
(__pthread_cond_broadcast): Rename __lll_mutex_* to __lll_*,
pass LLL_SHARED in %esi to both __lll_lock_wait and
__lll_unlock_wake.
* sysdeps/unix/sysv/linux/x86_64/pthread_cond_signal.S:
Include lowlevellock.h and pthread-errnos.h.
(SYS_futex, FUTEX_WAIT, FUTEX_WAKE, FUTEX_WAKE_OP,
FUTEX_OP_CLEAR_WAKE_IF_GT_ONE, EINVAL, LOCK): Don't define.
(__pthread_cond_signal): Rename __lll_mutex_* to __lll_*,
pass LLL_SHARED in %esi to both __lll_lock_wait and
__lll_unlock_wake.
* sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:
Include lowlevellock.h.
(SYS_futex, FUTEX_WAIT, FUTEX_WAKE, LOCK): Don't define.
(__pthread_cond_timedwait): Rename __lll_mutex_* to __lll_*,
pass LLL_SHARED in %esi to both __lll_lock_wait and
__lll_unlock_wake.
* sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:
Include lowlevellock.h.
(SYS_futex, FUTEX_WAIT, FUTEX_WAKE, LOCK): Don't define.
(__pthread_cond_wait, __condvar_cleanup): Rename __lll_mutex_*
to __lll_*, pass LLL_SHARED in %esi to both __lll_lock_wait
and __lll_unlock_wake.
* sysdeps/unix/sysv/linux/x86_64/pthread_rwlock_rdlock.S:
Include lowlevellock.h.
(SYS_futex, FUTEX_WAIT, FUTEX_WAKE, FUTEX_PRIVATE_FLAG, LOCK):
Don't define.
(__pthread_rwlock_rdlock): Rename __lll_mutex_* to __lll_*,
pass PSHARED(%rdi) in %esi to both __lll_lock_wait
and __lll_unlock_wake.
* sysdeps/unix/sysv/linux/x86_64/pthread_rwlock_timedrdlock.S:
Include lowlevellock.h.
(SYS_futex, FUTEX_WAIT, FUTEX_WAKE, FUTEX_PRIVATE_FLAG, LOCK):
Don't define.
(__pthread_rwlock_wrlock): Rename __lll_mutex_* to __lll_*,
pass PSHARED(%rdi) in %esi to both __lll_lock_wait
and __lll_unlock_wake.
* sysdeps/unix/sysv/linux/x86_64/pthread_rwlock_timedwrlock.S:
Include lowlevellock.h.
(SYS_futex, FUTEX_WAIT, FUTEX_WAKE, FUTEX_PRIVATE_FLAG, LOCK):
Don't define.
(__pthread_rwlock_wrlock): Rename __lll_mutex_* to __lll_*,
pass PSHARED(%rdi) in %esi to both __lll_lock_wait
and __lll_unlock_wake.
* sysdeps/unix/sysv/linux/x86_64/pthread_rwlock_unlock.S:
Include lowlevellock.h.
(SYS_futex, FUTEX_WAIT, FUTEX_WAKE, FUTEX_PRIVATE_FLAG, LOCK):
Don't define.
(__pthread_rwlock_unlock): Rename __lll_mutex_* to __lll_*,
pass PSHARED(%rdi) in %esi to both __lll_lock_wait
and __lll_unlock_wake.
* sysdeps/unix/sysv/linux/x86_64/pthread_rwlock_wrlock.S:
Include lowlevellock.h.
(SYS_futex, FUTEX_WAIT, FUTEX_WAKE, FUTEX_PRIVATE_FLAG, LOCK):
Don't define.
(__pthread_rwlock_wrlock): Rename __lll_mutex_* to __lll_*,
pass PSHARED(%rdi) in %ecx to both __lll_lock_wait
and __lll_unlock_wake.
* sysdeps/unix/sysv/linux/x86_64/pthread_once.S: Include
lowlevellock.h.
(LOCK, SYS_futex, FUTEX_WAIT, FUTEX_WAKE, FUTEX_PRIVATE_FLAG): Don't
define.
* sysdeps/unix/sysv/linux/x86_64/sem_post.S: Include lowlevellock.h.
(LOCK, SYS_futex, FUTEX_WAKE): Don't define.
* sysdeps/unix/sysv/linux/x86_64/sem_timedwait.S: Include
lowlevellock.h.
(LOCK, SYS_futex, FUTEX_WAIT): Don't define.
* sysdeps/unix/sysv/linux/x86_64/sem_trywait.S: Include
lowlevellock.h.
(LOCK): Don't define.
* sysdeps/unix/sysv/linux/x86_64/sem_wait.S: Include
lowlevellock.h.
(LOCK, SYS_futex, FUTEX_WAIT): Don't define.
2007-07-24 Jakub Jelinek <jakub@redhat.com>
* allocatestack.c (__nptl_setxid, __wait_lookup_done): Replace
lll_private_futex_* (*) with lll_futex_* (*, LLL_PRIVATE).
* pthread_create.c (start_thread): Likewise.
* init.c (sighandler_setxid): Likewise.
* sysdeps/ia64/tls.h (THREAD_GSCOPE_RESET_FLAG): Likewise.
* sysdeps/i386/tls.h (THREAD_GSCOPE_RESET_FLAG): Likewise.
* sysdeps/s390/tls.h (THREAD_GSCOPE_RESET_FLAG): Likewise.
* sysdeps/powerpc/tls.h (THREAD_GSCOPE_RESET_FLAG): Likewise.
* sysdeps/x86_64/tls.h (THREAD_GSCOPE_RESET_FLAG): Likewise.
* sysdeps/pthread/aio_misc.h (AIO_MISC_NOTIFY, AIO_MISC_WAIT):
Likewise.
* sysdeps/pthread/gai_misc.h (GAI_MISC_NOTIFY, GAI_MISC_WAIT):
Likewise.
* sysdeps/unix/sysv/linux/unregister-atfork.c (__unregister_atfork):
Likewise.
* sysdeps/unix/sysv/linux/fork.c (__libc_fork): Likewise.
* sysdeps/unix/sysv/linux/powerpc/pthread_once.c (clear_once_control,
__pthread_once): Likewise.
* sysdeps/unix/sysv/linux/ia64/pthread_once.c (clear_once_control,
__pthread_once): Add LLL_PRIVATE as last argument to lll_futex_*.
* sysdeps/unix/sysv/linux/ia64/lowlevellock.h (FUTEX_PRIVATE_FLAG,
LLL_PRIVATE, LLL_SHARED, __lll_private_flag): Define.
(lll_futex_wait): Add private argument, define as wrapper around
lll_futex_timed_wait.
(lll_futex_timed_wait, lll_futex_wake): Add private argument,
use __lll_private_flag macro.
(__lll_mutex_unlock, __lll_robust_mutex_unlock, lll_wait_tid,
__lll_mutex_unlock_force): Pass LLL_SHARED as last arg to lll_futex_*.
* sysdeps/unix/sysv/linux/i386/lowlevellock.h (__lll_private_flag):
Define.
(lll_futex_timed_wait, lll_futex_wake): Use it.
(lll_private_futex_wait, lll_private_futex_timed_wait,
lll_private_futex_wake): Removed.
* sysdeps/unix/sysv/linux/s390/pthread_once.c (clear_once_control,
__pthread_once): Add LLL_PRIVATE as last argument to lll_futex_*.
* sysdeps/unix/sysv/linux/s390/lowlevellock.h (FUTEX_PRIVATE_FLAG,
LLL_PRIVATE, LLL_SHARED, __lll_private_flag): Define.
(lll_futex_wait): Add private argument, define as wrapper around
lll_futex_timed_wait.
(lll_futex_timed_wait, lll_futex_wake): Add private argument,
use __lll_private_flag macro.
(lll_robust_mutex_dead, __lll_mutex_unlock, __lll_robust_mutex_unlock,
lll_wait_tid, __lll_mutex_unlock_force): Pass LLL_SHARED as last arg
to lll_futex_*.
* sysdeps/unix/sysv/linux/powerpc/lowlevellock.h
(lll_private_futex_wait, lll_private_futex_timed_wait,
lll_private_futex_wake): Removed.
* sysdeps/unix/sysv/linux/x86_64/lowlevellock.h (__lll_private_flag):
Fix !__ASSUME_PRIVATE_FUTEX non-constant private case.
(lll_private_futex_wait, lll_private_futex_timed_wait,
lll_private_futex_wake): Removed.
2007-07-23 Ulrich Drepper <drepper@redhat.com>
* sysdeps/unix/sysv/linux/x86_64/pthread_rwlock_timedrdlock.S: Fix
code used when private futexes are assumed.
* sysdeps/unix/sysv/linux/x86_64/pthread_rwlock_timedwrlock.S:
Likewise.
2007-07-23 Jakub Jelinek <jakub@redhat.com>
* sysdeps/unix/sysv/linux/powerpc/lowlevellock.h
(__lll_private_flag): Define.
(lll_futex_wait): Define as a wrapper around lll_futex_timed_wait.
(lll_futex_timed_wait, lll_futex_wake, lll_futex_wake_unlock): Use
__lll_private_flag.
(lll_private_futex_wait, lll_private_futex_timedwait,
lll_private_futex_wake): Define as wrapper around non-_private
macros.
* sysdeps/unix/sysv/linux/x86_64/lowlevellock.h
(__lll_private_flag): Define.
(lll_futex_timed_wait, lll_futex_wake): Use __lll_private_flag.
(lll_private_futex_wait, lll_private_futex_timedwait,
lll_private_futex_wake): Define as wrapper around non-_private
macros.
2007-07-10 Steven Munroe <sjmunroe@us.ibm.com>
* pthread_rwlock_rdlock.c (__pthread_rwlock_rdlock): Add LLL_SHARED
parameter to lll_futex_wait call.
* pthread_rwlock_wrlock.c (__pthread_rwlock_wrlock): Likewise.
* sysdeps/unix/sysv/linux/powerpc/pthread_once.c (__pthread_once):
Replace lll_futex_wait with lll_private_futex_wait.
* sysdeps/unix/sysv/linux/powerpc/sem_post.c (__new_sem_post):
Add LLL_SHARED parameter to lll_futex_wake().
* sysdeps/unix/sysv/linux/powerpc/lowlevellock.h: Define LLL_PRIVATE
LLL_SHARED, lll_private_futex_wait, lll_private_futex_timed_wait and
lll_private_futex_wake.
(lll_futex_wait): Add private parameter. Adjust FUTEX_PRIVATE_FLAG
bit from private parm before syscall.
(lll_futex_timed_wait): Likewise.
(lll_futex_wake): Likewise.
(lll_futex_wake_unlock): Likewise.
(lll_mutex_unlock): Add LLL_SHARED parm to lll_futex_wake call.
(lll_robust_mutex_unlock): Likewise.
(lll_mutex_unlock_force): Likewise.
(lll_wait_tid): Add LLL_SHARED parm to lll_futex_wait call.
2007-07-23 Ulrich Drepper <drepper@redhat.com>
* sysdeps/unix/sysv/linux/x86_64/pthread_rwlock_timedrdlock.S: Fix
compilation when unconditionally using private futexes.
* sysdeps/unix/sysv/linux/x86_64/pthread_rwlock_rdlock.S: Likewise.
* sysdeps/unix/sysv/linux/x86_64/pthread_rwlock_timedwrlock.S:
Likewise.
* sysdeps/unix/sysv/linux/x86_64/pthread_rwlock_unlock.S: Likewise.
* sysdeps/unix/sysv/linux/x86_64/pthread_rwlock_wrlock.S: Likewise.
2007-06-18 Ulrich Drepper <drepper@redhat.com>
* pthreadP.h: Define PTHREAD_MUTEX_TYPE.
* phtread_mutex_lock.c: Use PTHREAD_MUTEX_TYPE.
* pthread_mutex_timedlock.c: Likewise.
* pthread_mutex_trylock.c: Likewise.
* pthread_mutex_unlock.c: Likewise.
2007-06-17 Ulrich Drepper <drepper@redhat.com>
* allocatestack.c (allocate_stack): Make code compile if
__ASSUME_PRIVATE_FUTEX is set.
2007-06-07 Ulrich Drepper <drepper@redhat.com>
* sysdeps/unix/sysv/linux/x86_64/lowlevellock.h: Add additional
parameter to lll_futex_wait, lll_futex_timed_wait, and
lll_futex_wake. Change lll_futex_wait to call lll_futex_timed_wait.
Add lll_private_futex_wait, lll_private_futex_timed_wait, and
lll_private_futex_wake.
* sysdeps/unix/sysv/linux/i386/lowlevellock.h: Likewise.
* allocatestack.c: Adjust use of lll_futex_* macros.
* init.c: Likewise.
* lowlevellock.h: Likewise.
* pthread_barrier_wait.c: Likewise.
* pthread_cond_broadcast.c: Likewise.
* pthread_cond_destroy.c: Likewise.
* pthread_cond_signal.c: Likewise.
* pthread_cond_timedwait.c: Likewise.
* pthread_cond_wait.c: Likewise.
* pthread_create.c: Likewise.
* pthread_mutex_lock.c: Likewise.
* pthread_mutex_setprioceiling.c: Likewise.
* pthread_mutex_timedlock.c: Likewise.
* pthread_mutex_unlock.c: Likewise.
* pthread_rwlock_timedrdlock.c: Likewise.
* pthread_rwlock_timedwrlock.c: Likewise.
* pthread_rwlock_unlock.c: Likewise.
* sysdeps/i386/tls.h: Likewise.
* sysdeps/ia64/tls.h: Likewise.
* sysdeps/powerpc/tls.h: Likewise.
* sysdeps/pthread/aio_misc.h: Likewise.
* sysdeps/pthread/gai_misc.h: Likewise.
* sysdeps/s390/tls.h: Likewise.
* sysdeps/unix/sysv/linux/fork.c: Likewise.
* sysdeps/unix/sysv/linux/lowlevellock.c: Likewise.
* sysdeps/unix/sysv/linux/lowlevelrobustlock.c: Likewise.
* sysdeps/unix/sysv/linux/sem_post.c: Likewise.
* sysdeps/unix/sysv/linux/sem_timedwait.c: Likewise.
* sysdeps/unix/sysv/linux/sem_wait.c: Likewise.
* sysdeps/unix/sysv/linux/unregister-atfork.c: Likewise.
* sysdeps/x86_64/tls.h: Likewise.
2007-05-29 Ulrich Drepper <drepper@redhat.com>
* pthread_getattr_np.c: No need to install a cancellation handler,
this is no cancellation point.
* pthread_getschedparam.c: Likewise.
* pthread_setschedparam.c: Likewise.
* pthread_setschedprio.c: Likewise.
* sysdeps/unix/sysv/linux/lowlevellock.c: Remove all traces of
lll_unlock_wake_cb.
* sysdeps/unix/sysv/linux/i386/lowlevellock.h: Likewise.
* sysdeps/unix/sysv/linux/i386/i486/lowlevellock.S: Likewise.
* sysdeps/unix/sysv/linux/ia64/lowlevellock.h: Likewise.
* sysdeps/unix/sysv/linux/powerpc/lowlevellock.h: Likewise.
* sysdeps/unix/sysv/linux/s390/lowlevellock.h: Likewise.
* sysdeps/unix/sysv/linux/x86_64/lowlevellock.S: Likewise.
* sysdeps/unix/sysv/linux/x86_64/lowlevellock.h: Likewise.
* sysdeps/unix/sysv/linux/i386/i486/libc-lowlevellock.S: Checking
whether there are more than one thread makes no sense here since
we only call the slow path if the locks are taken.
* sysdeps/unix/sysv/linux/x86_64/libc-lowlevellock.S: Likewise.
* sysdeps/unix/sysv/linux/internaltypes.h: Introduce
COND_NWAITERS_SHIFT.
* pthread_cond_destroy.c: Use COND_NWAITERS_SHIFT instead of
COND_CLOCK_BITS.
* pthread_cond_init.c: Likewise.
* pthread_cond_timedwait.c: Likewise.
* pthread_cond_wait.c: Likewise.
* pthread_condattr_getclock.c: Likewise.
* pthread_condattr_setclock.c: Likewise.
* sysdeps/unix/sysv/linux/lowlevelcond.sym: Likewise.
* sysdeps/unix/sysv/linux/i386/i486/pthread_cond_timedwait.S: Likewise.
* sysdeps/unix/sysv/linux/i386/i486/pthread_cond_wait.S: Likewise.
* sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S: Likewise.
* sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S: Likewise.
2007-05-24 Richard Henderson <rth@redhat.com>
* descr.h (struct pthread): Add header.gscope_flag.
2007-05-27 Ulrich Drepper <drepper@redhat.com>
* init.c: Make it compile with older kernel headers.
* pthread_rwlock_init.c: Also initialize __shared field.
* sysdeps/unix/sysv/linux/i386/bits/pthreadtypes.h: Split __flags
element in rwlock structure into four byte elements. One of them is
the new __shared element.
* sysdeps/unix/sysv/linux/x86_64/bits/pthreadtypes.h [__WORDSIZE=32]:
Likewise.
[__WORDSIZE=64]: Renamed __pad1 element int rwlock structure to
__shared, adjust names of other padding elements.
* sysdeps/unix/sysv/linux/powerpc/bits/pthreadtypes.h: Likewise.
* sysdeps/pthread/pthread.h: Adjust rwlock initializers.
* sysdeps/unix/sysv/linux/lowlevelrwlock.sym: Add PSHARED.
* sysdeps/unix/sysv/linux/powerpc/lowlevellock.h: Define
FUTEX_PRIVATE_FLAG.
* sysdeps/unix/sysv/linux/x86_64/pthread_rwlock_rdlock.S: Change main
futex to use private operations if possible.
* sysdeps/unix/sysv/linux/x86_64/pthread_rwlock_timedrdlock.S:
Likewise.
* sysdeps/unix/sysv/linux/x86_64/pthread_rwlock_timedwrlock.S:
Likewise.
* sysdeps/unix/sysv/linux/x86_64/pthread_rwlock_unlock.S: Likewise.
* sysdeps/unix/sysv/linux/x86_64/pthread_rwlock_wrlock.S: Likewise.
* sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_rdlock.S: Likewise.
* sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_timedrdlock.S:
Likewise.
* sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_timedwrlock.S:
Likewise.
* sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_unlock.S: Likewise.
* sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_wrlock.S: Likewise.
2007-05-26 Ulrich Drepper <drepper@redhat.com>
* sysdeps/unix/sysv/linux/x86_64/sem_trywait.S (sem_trywait): Tiny
optimization.
* sysdeps/unix/sysv/linux/sem_wait.c: Add missing break.
* sysdeps/unix/sysv/linux/sem_timedwait.c: Removed left-over
duplication of __sem_wait_cleanup.
* allocatestack.c: Revert last change.
* init.c: Likewise.
* sysdeps/i386/tls.h: Likewise.
* sysdeps/x86_64/tls.h: Likewise.
* descr.h [TLS_DTV_AT_TP] (struct pthread): Add private_futex field to
header structure.
* sysdeps/powerpc/tcb-offsets.sym: Add PRIVATE_FUTEX_OFFSET.
* sysdeps/unix/sysv/linux/internaltypes.h (struct pthread_barrier):
Add private field.
* sysdeps/unix/sysv/linux/lowlevelbarrier.sym: Add PRIVATE definition.
* pthread_barrier_init.c: Set private flag if pshared and private
futexes are supported.
* sysdeps/unix/sysv/linux/i386/i486/pthread_barrier_wait.S: Use
private field in futex command setup.
* sysdeps/unix/sysv/linux/x86_64/pthread_barrier_wait.S: Likewise.
2007-05-25 Ulrich Drepper <drepper@redhat.com>
* sysdeps/unix/sysv/linux/i386/i486/sem_post.S: Add private futex
support.
* sysdeps/unix/sysv/linux/i386/i486/sem_timedwait.S: Likewise.
* sysdeps/unix/sysv/linux/i386/i486/sem_wait.S: Likewise.
* sysdeps/unix/sysv/linux/x86_64/sem_post.S: Likewise.
* sysdeps/unix/sysv/linux/x86_64/sem_timedwait.S: Likewise.
* sysdeps/unix/sysv/linux/x86_64/sem_wait.S: Likewise.
* semaphoreP.h: Declare __old_sem_init and __old_sem_wait.
* sem_init.c (__new_sem_init): Rewrite to initialize all three
fields in the structure.
(__old_sem_init): New function.
* sem_open.c: Initialize all fields of the structure.
* sem_getvalue.c: Adjust for renamed element.
* sysdeps/unix/sysv/linux/Makefile [subdir=nptl]
(gen-as-const-headers): Add structsem.sym.
* sysdeps/unix/sysv/linux/structsem.sym: New file.
* sysdeps/unix/sysv/linux/internaltypes.h: Rename struct sem to
struct new_sem. Add struct old_sem.
* sysdeps/unix/sysv/linux/sem_post.c: Wake only when there are waiters.
* sysdeps/unix/sysv/linux/i386/i486/sem_post.S: Likewise.
* sysdeps/unix/sysv/linux/x86_64/sem_post.S: Likewise.
* sysdeps/unix/sysv/linux/sem_wait.c: Indicate that there are waiters.
* sysdeps/unix/sysv/linux/i386/i486/sem_wait.S: Likewise.
* sysdeps/unix/sysv/linux/x86_64/sem_wait.S: Likewise.
* sysdeps/unix/sysv/linux/sem_timedwait.c: Likewise.
* sysdeps/unix/sysv/linux/i386/i486/sem_timedwait.S: Likewise.
* sysdeps/unix/sysv/linux/x86_64/sem_timedwait.S: Likewise.
* Makefile (tests): Add tst-sem11, tst-sem12.
* tst-sem11.c: New file.
* tst-sem12.c: New file.
* tst-typesizes.c: Test struct new_sem and struct old_sem instead
of struct sem.
2007-05-25 Ulrich Drepper <drepper@redhat.com>
Jakub Jelinek <jakub@redhat.com>
* sysdeps/unix/sysv/linux/i386/i486/sem_timedwait.S (sem_timedwait):
Move __pthread_enable_asynccancel right before futex syscall.
* sysdeps/unix/sysv/linux/x86_64/sem_timedwait.S (sem_timedwait):
Likewise.
2007-05-24 Jakub Jelinek <jakub@redhat.com>
* sysdeps/i386/tls.h (THREAD_SET_PRIVATE_FUTEX,
THREAD_COPY_PRIVATE_FUTEX): Define.
* sysdeps/x86_64/tls.h (THREAD_SET_PRIVATE_FUTEX,
THREAD_COPY_PRIVATE_FUTEX): Define.
* allocatestack.c (allocate_stack): Use THREAD_COPY_PRIVATE_FUTEX.
* init.c (__pthread_initialize_minimal_internal): Use
THREAD_SET_PRIVATE_FUTEX.
2007-05-23 Ulrich Drepper <drepper@redhat.com>
* init.c (__pthread_initialize_minimal_internal): Check whether
private futexes are available.
* allocatestack.c (allocate_stack): Copy private_futex field from
current thread into the new stack.
* sysdeps/unix/sysv/linux/x86_64/libc-lowlevellock.S: Use private
futexes if they are available.
* sysdeps/unix/sysv/linux/i386/i486/libc-lowlevellock.S: Likewise
* sysdeps/unix/sysv/linux/x86_64/lowlevellock.S: Adjust so that change
in libc-lowlevellock.S allow using private futexes.
* sysdeps/unix/sysv/linux/i386/i486/lowlevellock.S: Likewise.
* sysdeps/unix/sysv/linux/x86_64/lowlevellock.h: Define
FUTEX_PRIVATE_FLAG.
* sysdeps/unix/sysv/linux/i386/lowlevellock.h: Likewise.
* sysdeps/unix/sysv/linux/x86_64/pthread_once.S: Use private futexes
if they are available.
* sysdeps/unix/sysv/linux/i386/pthread_once.S: Likewise.
* sysdeps/x86_64/tcb-offsets.sym: Add PRIVATE_FUTEX.
* sysdeps/i386/tcb-offsets.sym: Likewise.
* sysdeps/x86_64/tls.h (tcbhead_t): Add private_futex field.
* sysdeps/i386/tls.h (tcbhead_t): Likewise.
2007-05-17 Ulrich Drepper <drepper@redhat.com>
* sysdeps/unix/sysv/linux/sem_wait.c (__new_sem_wait): Remove
unnecessary extra cancellation test.
2007-05-14 Ulrich Drepper <drepper@redhat.com>
* sysdeps/unix/sysv/linux/x86_64/sem_wait.S: Remove unnecessary
extra cancellation test.
* sysdeps/unix/sysv/linux/x86_64/sem_timedwait.S: Likewise.
2007-01-10 Ulrich Drepper <drepper@redhat.com>
* sysdeps/unix/sysv/linux/x86_64/lowlevellock.h: Cleanups. Define
FUTEX_CMP_REQUEUE and lll_futex_requeue.
--- libc/nptl/Makefile 18 May 2007 00:52:02 -0000 1.189
+++ libc/nptl/Makefile 21 Aug 2007 23:54:26 -0000 1.191
@@ -218,7 +218,7 @@ tests = tst-typesizes \
tst-once1 tst-once2 tst-once3 tst-once4 \
tst-key1 tst-key2 tst-key3 tst-key4 \
tst-sem1 tst-sem2 tst-sem3 tst-sem4 tst-sem5 tst-sem6 tst-sem7 \
- tst-sem8 tst-sem9 tst-sem10 \
+ tst-sem8 tst-sem9 tst-sem10 tst-sem11 tst-sem12 \
tst-barrier1 tst-barrier2 tst-barrier3 tst-barrier4 \
tst-align tst-align2 tst-align3 \
tst-basic1 tst-basic2 tst-basic3 tst-basic4 tst-basic5 tst-basic6 \
--- libc/nptl/allocatestack.c 19 May 2007 07:05:45 -0000 1.66
+++ libc/nptl/allocatestack.c 21 Aug 2007 23:55:08 -0000 1.78
@@ -28,6 +28,7 @@
#include <dl-sysdep.h>
#include <tls.h>
#include <lowlevellock.h>
+#include <kernel-features.h>
#ifndef NEED_SEPARATE_REGISTER_STACK
@@ -103,7 +104,7 @@ static size_t stack_cache_maxsize = 40 *
static size_t stack_cache_actsize;
/* Mutex protecting this variable. */
-static lll_lock_t stack_cache_lock = LLL_LOCK_INITIALIZER;
+static int stack_cache_lock = LLL_LOCK_INITIALIZER;
/* List of queued stack frames. */
static LIST_HEAD (stack_cache);
@@ -139,7 +140,7 @@ get_cached_stack (size_t *sizep, void **
struct pthread *result = NULL;
list_t *entry;
- lll_lock (stack_cache_lock);
+ lll_lock (stack_cache_lock, LLL_PRIVATE);
/* Search the cache for a matching entry. We search for the
smallest stack which has at least the required size. Note that
@@ -172,7 +173,7 @@ get_cached_stack (size_t *sizep, void **
|| __builtin_expect (result->stackblock_size > 4 * size, 0))
{
/* Release the lock. */
- lll_unlock (stack_cache_lock);
+ lll_unlock (stack_cache_lock, LLL_PRIVATE);
return NULL;
}
@@ -187,7 +188,7 @@ get_cached_stack (size_t *sizep, void **
stack_cache_actsize -= result->stackblock_size;
/* Release the lock early. */
- lll_unlock (stack_cache_lock);
+ lll_unlock (stack_cache_lock, LLL_PRIVATE);
/* Report size and location of the stack to the caller. */
*sizep = result->stackblock_size;
@@ -371,6 +372,12 @@ allocate_stack (const struct pthread_att
__pthread_multiple_threads = *__libc_multiple_threads_ptr = 1;
#endif
+#ifndef __ASSUME_PRIVATE_FUTEX
+ /* The thread must know when private futexes are supported. */
+ pd->header.private_futex = THREAD_GETMEM (THREAD_SELF,
+ header.private_futex);
+#endif
+
#ifdef NEED_DL_SYSINFO
/* Copy the sysinfo value from the parent. */
THREAD_SYSINFO(pd) = THREAD_SELF_SYSINFO;
@@ -389,12 +396,12 @@ allocate_stack (const struct pthread_att
/* Prepare to modify global data. */
- lll_lock (stack_cache_lock);
+ lll_lock (stack_cache_lock, LLL_PRIVATE);
/* And add to the list of stacks in use. */
list_add (&pd->list, &__stack_user);
- lll_unlock (stack_cache_lock);
+ lll_unlock (stack_cache_lock, LLL_PRIVATE);
}
else
{
@@ -505,6 +512,12 @@ allocate_stack (const struct pthread_att
__pthread_multiple_threads = *__libc_multiple_threads_ptr = 1;
#endif
+#ifndef __ASSUME_PRIVATE_FUTEX
+ /* The thread must know when private futexes are supported. */
+ pd->header.private_futex = THREAD_GETMEM (THREAD_SELF,
+ header.private_futex);
+#endif
+
#ifdef NEED_DL_SYSINFO
/* Copy the sysinfo value from the parent. */
THREAD_SYSINFO(pd) = THREAD_SELF_SYSINFO;
@@ -527,12 +540,12 @@ allocate_stack (const struct pthread_att
/* Prepare to modify global data. */
- lll_lock (stack_cache_lock);
+ lll_lock (stack_cache_lock, LLL_PRIVATE);
/* And add to the list of stacks in use. */
list_add (&pd->list, &stack_used);
- lll_unlock (stack_cache_lock);
+ lll_unlock (stack_cache_lock, LLL_PRIVATE);
/* There might have been a race. Another thread might have
@@ -579,12 +592,12 @@ allocate_stack (const struct pthread_att
mprot_error:
err = errno;
- lll_lock (stack_cache_lock);
+ lll_lock (stack_cache_lock, LLL_PRIVATE);
/* Remove the thread from the list. */
list_del (&pd->list);
- lll_unlock (stack_cache_lock);
+ lll_unlock (stack_cache_lock, LLL_PRIVATE);
/* Get rid of the TLS block we allocated. */
_dl_deallocate_tls (TLS_TPADJ (pd), false);
@@ -673,7 +686,7 @@ void
internal_function
__deallocate_stack (struct pthread *pd)
{
- lll_lock (stack_cache_lock);
+ lll_lock (stack_cache_lock, LLL_PRIVATE);
/* Remove the thread from the list of threads with user defined
stacks. */
@@ -689,7 +702,7 @@ __deallocate_stack (struct pthread *pd)
/* Free the memory associated with the ELF TLS. */
_dl_deallocate_tls (TLS_TPADJ (pd), false);
- lll_unlock (stack_cache_lock);
+ lll_unlock (stack_cache_lock, LLL_PRIVATE);
}
@@ -706,7 +719,7 @@ __make_stacks_executable (void **stack_e
const size_t pagemask = ~(__getpagesize () - 1);
#endif
- lll_lock (stack_cache_lock);
+ lll_lock (stack_cache_lock, LLL_PRIVATE);
list_t *runp;
list_for_each (runp, &stack_used)
@@ -735,7 +748,7 @@ __make_stacks_executable (void **stack_e
break;
}
- lll_unlock (stack_cache_lock);
+ lll_unlock (stack_cache_lock, LLL_PRIVATE);
return err;
}
@@ -811,7 +824,7 @@ __find_thread_by_id (pid_t tid)
{
struct pthread *result = NULL;
- lll_lock (stack_cache_lock);
+ lll_lock (stack_cache_lock, LLL_PRIVATE);
/* Iterate over the list with system-allocated threads first. */
list_t *runp;
@@ -843,7 +856,7 @@ __find_thread_by_id (pid_t tid)
}
out:
- lll_unlock (stack_cache_lock);
+ lll_unlock (stack_cache_lock, LLL_PRIVATE);
return result;
}
@@ -894,7 +907,7 @@ attribute_hidden
__nptl_setxid (struct xid_command *cmdp)
{
int result;
- lll_lock (stack_cache_lock);
+ lll_lock (stack_cache_lock, LLL_PRIVATE);
__xidcmd = cmdp;
cmdp->cntr = 0;
@@ -925,7 +938,7 @@ __nptl_setxid (struct xid_command *cmdp)
int cur = cmdp->cntr;
while (cur != 0)
{
- lll_futex_wait (&cmdp->cntr, cur);
+ lll_futex_wait (&cmdp->cntr, cur, LLL_PRIVATE);
cur = cmdp->cntr;
}
@@ -940,7 +953,7 @@ __nptl_setxid (struct xid_command *cmdp)
result = -1;
}
- lll_unlock (stack_cache_lock);
+ lll_unlock (stack_cache_lock, LLL_PRIVATE);
return result;
}
@@ -969,7 +982,7 @@ void
attribute_hidden
__pthread_init_static_tls (struct link_map *map)
{
- lll_lock (stack_cache_lock);
+ lll_lock (stack_cache_lock, LLL_PRIVATE);
/* Iterate over the list with system-allocated threads first. */
list_t *runp;
@@ -980,7 +993,7 @@ __pthread_init_static_tls (struct link_m
list_for_each (runp, &__stack_user)
init_one_static_tls (list_entry (runp, struct pthread, list), map);
- lll_unlock (stack_cache_lock);
+ lll_unlock (stack_cache_lock, LLL_PRIVATE);
}
@@ -988,7 +1001,7 @@ void
attribute_hidden
__wait_lookup_done (void)
{
- lll_lock (stack_cache_lock);
+ lll_lock (stack_cache_lock, LLL_PRIVATE);
struct pthread *self = THREAD_SELF;
@@ -1011,7 +1024,7 @@ __wait_lookup_done (void)
continue;
do
- lll_futex_wait (gscope_flagp, THREAD_GSCOPE_FLAG_WAIT);
+ lll_futex_wait (gscope_flagp, THREAD_GSCOPE_FLAG_WAIT, LLL_PRIVATE);
while (*gscope_flagp == THREAD_GSCOPE_FLAG_WAIT);
}
@@ -1033,9 +1046,9 @@ __wait_lookup_done (void)
continue;
do
- lll_futex_wait (gscope_flagp, THREAD_GSCOPE_FLAG_WAIT);
+ lll_futex_wait (gscope_flagp, THREAD_GSCOPE_FLAG_WAIT, LLL_PRIVATE);
while (*gscope_flagp == THREAD_GSCOPE_FLAG_WAIT);
}
- lll_unlock (stack_cache_lock);
+ lll_unlock (stack_cache_lock, LLL_PRIVATE);
}
--- libc/nptl/descr.h 11 May 2007 06:11:48 -0000 1.39
+++ libc/nptl/descr.h 1 Aug 2007 04:09:32 -0000 1.42
@@ -37,6 +37,7 @@
#endif
#define __need_res_state
#include <resolv.h>
+#include <kernel-features.h>
#ifndef TCB_ALIGNMENT
# define TCB_ALIGNMENT sizeof (double)
@@ -132,6 +133,9 @@ struct pthread
{
int multiple_threads;
int gscope_flag;
+# ifndef __ASSUME_PRIVATE_FUTEX
+ int private_futex;
+# endif
} header;
#endif
@@ -302,10 +306,10 @@ struct pthread
int parent_cancelhandling;
/* Lock to synchronize access to the descriptor. */
- lll_lock_t lock;
+ int lock;
/* Lock for synchronizing setxid calls. */
- lll_lock_t setxid_futex;
+ int setxid_futex;
#if HP_TIMING_AVAIL
/* Offset of the CPU clock at start thread start time. */
--- libc/nptl/init.c 21 May 2007 22:29:51 -0000 1.62
+++ libc/nptl/init.c 15 Oct 2007 20:25:25 -0000 1.74
@@ -33,6 +33,7 @@
#include <shlib-compat.h>
#include <smp.h>
#include <lowlevellock.h>
+#include <kernel-features.h>
#ifndef __NR_set_tid_address
@@ -237,7 +238,7 @@ sighandler_setxid (int sig, siginfo_t *s
__xidcmd->id[1], __xidcmd->id[2]);
if (atomic_decrement_val (&__xidcmd->cntr) == 0)
- lll_futex_wake (&__xidcmd->cntr, 1);
+ lll_futex_wake (&__xidcmd->cntr, 1, LLL_PRIVATE);
/* Reset the SETXID flag. */
struct pthread *self = THREAD_SELF;
@@ -246,7 +247,7 @@ sighandler_setxid (int sig, siginfo_t *s
/* And release the futex. */
self->setxid_futex = 1;
- lll_futex_wake (&self->setxid_futex, 1);
+ lll_futex_wake (&self->setxid_futex, 1, LLL_PRIVATE);
}
@@ -300,6 +301,18 @@ __pthread_initialize_minimal_internal (v
#endif
set_robust_list_not_avail ();
+#ifndef __ASSUME_PRIVATE_FUTEX
+ /* Private futexes are always used (at least internally) so that
+ doing the test once this early is beneficial. */
+ {
+ int word = 0;
+ word = INTERNAL_SYSCALL (futex, err, 3, &word,
+ FUTEX_WAKE | FUTEX_PRIVATE_FLAG, 1);
+ if (!INTERNAL_SYSCALL_ERROR_P (word, err))
+ THREAD_SETMEM (pd, header.private_futex, FUTEX_PRIVATE_FLAG);
+ }
+#endif
+
/* Set initial thread's stack block from 0 up to __libc_stack_end.
It will be bigger than it actually is, but for unwind.c/pt-longjmp.c
purposes this is good enough. */
--- libc/nptl/lowlevellock.h 27 Feb 2006 00:58:29 -0000 1.1
+++ libc/nptl/lowlevellock.h 8 Jun 2007 02:47:50 -0000 1.2
@@ -21,16 +21,6 @@
#include <atomic.h>
-/* Implement generic mutex. Basic futex syscall support is required:
-
- lll_futex_wait(futex, value) - call sys_futex with FUTEX_WAIT
- and third parameter VALUE
-
- lll_futex_wake(futex, value) - call sys_futex with FUTEX_WAKE
- and third parameter VALUE
-*/
-
-
/* Mutex lock counter:
bit 31 clear means unlocked;
bit 31 set means locked.
@@ -66,7 +56,9 @@ __generic_mutex_lock (int *mutex)
if (v >= 0)
continue;
- lll_futex_wait (mutex, v);
+ lll_futex_wait (mutex, v,
+ // XYZ check mutex flag
+ LLL_SHARED);
}
}
@@ -82,7 +74,9 @@ __generic_mutex_unlock (int *mutex)
/* There are other threads waiting for this mutex, wake one of them
up. */
- lll_futex_wake (mutex, 1);
+ lll_futex_wake (mutex, 1,
+ // XYZ check mutex flag
+ LLL_SHARED);
}
--- libc/nptl/old_pthread_cond_broadcast.c 21 Mar 2003 08:02:07 -0000 1.7
+++ libc/nptl/old_pthread_cond_broadcast.c 1 Aug 2007 04:13:12 -0000 1.8
@@ -33,7 +33,7 @@ __pthread_cond_broadcast_2_0 (cond)
{
pthread_cond_t *newcond;
-#if LLL_MUTEX_LOCK_INITIALIZER == 0
+#if LLL_LOCK_INITIALIZER == 0
newcond = (pthread_cond_t *) calloc (sizeof (pthread_cond_t), 1);
if (newcond == NULL)
return ENOMEM;
--- libc/nptl/old_pthread_cond_signal.c 21 Mar 2003 08:02:07 -0000 1.7
+++ libc/nptl/old_pthread_cond_signal.c 1 Aug 2007 04:09:02 -0000 1.8
@@ -33,7 +33,7 @@ __pthread_cond_signal_2_0 (cond)
{
pthread_cond_t *newcond;
-#if LLL_MUTEX_LOCK_INITIALIZER == 0
+#if LLL_LOCK_INITIALIZER == 0
newcond = (pthread_cond_t *) calloc (sizeof (pthread_cond_t), 1);
if (newcond == NULL)
return ENOMEM;
--- libc/nptl/old_pthread_cond_timedwait.c 21 Mar 2003 08:02:07 -0000 1.6
+++ libc/nptl/old_pthread_cond_timedwait.c 1 Aug 2007 04:09:21 -0000 1.7
@@ -35,7 +35,7 @@ __pthread_cond_timedwait_2_0 (cond, mute
{
pthread_cond_t *newcond;
-#if LLL_MUTEX_LOCK_INITIALIZER == 0
+#if LLL_LOCK_INITIALIZER == 0
newcond = (pthread_cond_t *) calloc (sizeof (pthread_cond_t), 1);
if (newcond == NULL)
return ENOMEM;
--- libc/nptl/old_pthread_cond_wait.c 21 Mar 2003 08:02:07 -0000 1.7
+++ libc/nptl/old_pthread_cond_wait.c 1 Aug 2007 04:11:12 -0000 1.8
@@ -34,7 +34,7 @@ __pthread_cond_wait_2_0 (cond, mutex)
{
pthread_cond_t *newcond;
-#if LLL_MUTEX_LOCK_INITIALIZER == 0
+#if LLL_LOCK_INITIALIZER == 0
newcond = (pthread_cond_t *) calloc (sizeof (pthread_cond_t), 1);
if (newcond == NULL)
return ENOMEM;
--- libc/nptl/pthreadP.h 19 May 2007 07:06:10 -0000 1.65
+++ libc/nptl/pthreadP.h 11 Aug 2007 18:45:00 -0000 1.69
@@ -96,6 +96,22 @@ enum
PTHREAD_MUTEX_PP_ADAPTIVE_NP
= PTHREAD_MUTEX_PRIO_PROTECT_NP | PTHREAD_MUTEX_ADAPTIVE_NP
};
+#define PTHREAD_MUTEX_PSHARED_BIT 128
+
+#define PTHREAD_MUTEX_TYPE(m) \
+ ((m)->__data.__kind & 127)
+
+#if LLL_PRIVATE == 0 && LLL_SHARED == 128
+# define PTHREAD_MUTEX_PSHARED(m) \
+ ((m)->__data.__kind & 128)
+#else
+# define PTHREAD_MUTEX_PSHARED(m) \
+ (((m)->__data.__kind & 128) ? LLL_SHARED : LLL_PRIVATE)
+#endif
+
+/* The kernel when waking robust mutexes on exit never uses
+ FUTEX_PRIVATE_FLAG FUTEX_WAKE. */
+#define PTHREAD_ROBUST_MUTEX_PSHARED(m) LLL_SHARED
/* Ceiling in __data.__lock. __data.__lock is signed, so don't
use the MSB bit in there, but in the mask also include that bit,
@@ -143,7 +159,7 @@ hidden_proto (__stack_user)
/* Attribute handling. */
extern struct pthread_attr *__attr_list attribute_hidden;
-extern lll_lock_t __attr_list_lock attribute_hidden;
+extern int __attr_list_lock attribute_hidden;
/* First available RT signal. */
extern int __current_sigrtmin attribute_hidden;
--- libc/nptl/pthread_attr_init.c 18 Mar 2004 23:56:31 -0000 1.4
+++ libc/nptl/pthread_attr_init.c 1 Aug 2007 04:14:30 -0000 1.5
@@ -27,7 +27,7 @@
struct pthread_attr *__attr_list;
-lll_lock_t __attr_list_lock = LLL_LOCK_INITIALIZER;
+int __attr_list_lock = LLL_LOCK_INITIALIZER;
int
--- libc/nptl/pthread_barrier_destroy.c 26 Nov 2002 22:49:50 -0000 1.1.1.1
+++ libc/nptl/pthread_barrier_destroy.c 1 Aug 2007 04:15:21 -0000 1.2
@@ -31,14 +31,14 @@ pthread_barrier_destroy (barrier)
ibarrier = (struct pthread_barrier *) barrier;
- lll_lock (ibarrier->lock);
+ lll_lock (ibarrier->lock, ibarrier->private ^ FUTEX_PRIVATE_FLAG);
if (__builtin_expect (ibarrier->left == ibarrier->init_count, 1))
/* The barrier is not used anymore. */
result = 0;
else
/* Still used, return with an error. */
- lll_unlock (ibarrier->lock);
+ lll_unlock (ibarrier->lock, ibarrier->private ^ FUTEX_PRIVATE_FLAG);
return result;
}
--- libc/nptl/pthread_barrier_init.c 26 Nov 2002 22:49:51 -0000 1.1.1.1
+++ libc/nptl/pthread_barrier_init.c 1 Aug 2007 04:10:54 -0000 1.3
@@ -20,6 +20,13 @@
#include <errno.h>
#include "pthreadP.h"
#include <lowlevellock.h>
+#include <kernel-features.h>
+
+
+static const struct pthread_barrierattr default_attr =
+ {
+ .pshared = PTHREAD_PROCESS_PRIVATE
+ };
int
@@ -33,17 +40,15 @@ pthread_barrier_init (barrier, attr, cou
if (__builtin_expect (count == 0, 0))
return EINVAL;
- if (attr != NULL)
- {
- struct pthread_barrierattr *iattr;
-
- iattr = (struct pthread_barrierattr *) attr;
-
- if (iattr->pshared != PTHREAD_PROCESS_PRIVATE
- && __builtin_expect (iattr->pshared != PTHREAD_PROCESS_SHARED, 0))
- /* Invalid attribute. */
- return EINVAL;
- }
+ const struct pthread_barrierattr *iattr
+ = (attr != NULL
+ ? iattr = (struct pthread_barrierattr *) attr
+ : &default_attr);
+
+ if (iattr->pshared != PTHREAD_PROCESS_PRIVATE
+ && __builtin_expect (iattr->pshared != PTHREAD_PROCESS_SHARED, 0))
+ /* Invalid attribute. */
+ return EINVAL;
ibarrier = (struct pthread_barrier *) barrier;
@@ -53,5 +58,14 @@ pthread_barrier_init (barrier, attr, cou
ibarrier->init_count = count;
ibarrier->curr_event = 0;
+#ifdef __ASSUME_PRIVATE_FUTEX
+ ibarrier->private = (iattr->pshared != PTHREAD_PROCESS_PRIVATE
+ ? 0 : FUTEX_PRIVATE_FLAG);
+#else
+ ibarrier->private = (iattr->pshared != PTHREAD_PROCESS_PRIVATE
+ ? 0 : THREAD_GETMEM (THREAD_SELF,
+ header.private_futex));
+#endif
+
return 0;
}
--- libc/nptl/sysdeps/pthread/pthread_barrier_wait.c 28 Oct 2006 05:06:42 -0000 1.1
+++ libc/nptl/sysdeps/pthread/pthread_barrier_wait.c 1 Aug 2007 04:18:50 -0000 1.3
@@ -32,7 +32,7 @@ pthread_barrier_wait (barrier)
int result = 0;
/* Make sure we are alone. */
- lll_lock (ibarrier->lock);
+ lll_lock (ibarrier->lock, ibarrier->private ^ FUTEX_PRIVATE_FLAG);
/* One more arrival. */
--ibarrier->left;
@@ -45,7 +45,8 @@ pthread_barrier_wait (barrier)
++ibarrier->curr_event;
/* Wake up everybody. */
- lll_futex_wake (&ibarrier->curr_event, INT_MAX);
+ lll_futex_wake (&ibarrier->curr_event, INT_MAX,
+ ibarrier->private ^ FUTEX_PRIVATE_FLAG);
/* This is the thread which finished the serialization. */
result = PTHREAD_BARRIER_SERIAL_THREAD;
@@ -57,11 +58,12 @@ pthread_barrier_wait (barrier)
unsigned int event = ibarrier->curr_event;
/* Before suspending, make the barrier available to others. */
- lll_unlock (ibarrier->lock);
+ lll_unlock (ibarrier->lock, ibarrier->private ^ FUTEX_PRIVATE_FLAG);
/* Wait for the event counter of the barrier to change. */
do
- lll_futex_wait (&ibarrier->curr_event, event);
+ lll_futex_wait (&ibarrier->curr_event, event,
+ ibarrier->private ^ FUTEX_PRIVATE_FLAG);
while (event == ibarrier->curr_event);
}
@@ -71,7 +73,7 @@ pthread_barrier_wait (barrier)
/* If this was the last woken thread, unlock. */
if (atomic_increment_val (&ibarrier->left) == init_count)
/* We are done. */
- lll_unlock (ibarrier->lock);
+ lll_unlock (ibarrier->lock, ibarrier->private ^ FUTEX_PRIVATE_FLAG);
return result;
}
--- libc/nptl/sysdeps/pthread/pthread_cond_broadcast.c 28 Oct 2006 05:07:08 -0000 1.4
+++ libc/nptl/sysdeps/pthread/pthread_cond_broadcast.c 11 Aug 2007 18:44:39 -0000 1.7
@@ -32,8 +32,10 @@ int
__pthread_cond_broadcast (cond)
pthread_cond_t *cond;
{
+ int pshared = (cond->__data.__mutex == (void *) ~0l)
+ ? LLL_SHARED : LLL_PRIVATE;
/* Make sure we are alone. */
- lll_mutex_lock (cond->__data.__lock);
+ lll_lock (cond->__data.__lock, pshared);
/* Are there any waiters to be woken? */
if (cond->__data.__total_seq > cond->__data.__wakeup_seq)
@@ -47,7 +49,7 @@ __pthread_cond_broadcast (cond)
++cond->__data.__broadcast_seq;
/* We are done. */
- lll_mutex_unlock (cond->__data.__lock);
+ lll_unlock (cond->__data.__lock, pshared);
/* Do not use requeue for pshared condvars. */
if (cond->__data.__mutex == (void *) ~0l)
@@ -57,19 +59,22 @@ __pthread_cond_broadcast (cond)
pthread_mutex_t *mut = (pthread_mutex_t *) cond->__data.__mutex;
/* XXX: Kernel so far doesn't support requeue to PI futex. */
- if (__builtin_expect (mut->__data.__kind & PTHREAD_MUTEX_PRIO_INHERIT_NP,
- 0))
+ /* XXX: Kernel so far can only requeue to the same type of futex,
+ in this case private (we don't requeue for pshared condvars). */
+ if (__builtin_expect (mut->__data.__kind
+ & (PTHREAD_MUTEX_PRIO_INHERIT_NP
+ | PTHREAD_MUTEX_PSHARED_BIT), 0))
goto wake_all;
/* lll_futex_requeue returns 0 for success and non-zero
for errors. */
if (__builtin_expect (lll_futex_requeue (&cond->__data.__futex, 1,
INT_MAX, &mut->__data.__lock,
- futex_val), 0))
+ futex_val, LLL_PRIVATE), 0))
{
/* The requeue functionality is not available. */
wake_all:
- lll_futex_wake (&cond->__data.__futex, INT_MAX);
+ lll_futex_wake (&cond->__data.__futex, INT_MAX, pshared);
}
/* That's all. */
@@ -77,7 +82,7 @@ __pthread_cond_broadcast (cond)
}
/* We are done. */
- lll_mutex_unlock (cond->__data.__lock);
+ lll_unlock (cond->__data.__lock, pshared);
return 0;
}
--- libc/nptl/pthread_cond_destroy.c 25 Sep 2005 23:58:11 -0000 1.6
+++ libc/nptl/pthread_cond_destroy.c 11 Aug 2007 18:43:43 -0000 1.10
@@ -26,14 +26,17 @@ int
__pthread_cond_destroy (cond)
pthread_cond_t *cond;
{
+ int pshared = (cond->__data.__mutex == (void *) ~0l)
+ ? LLL_SHARED : LLL_PRIVATE;
+
/* Make sure we are alone. */
- lll_mutex_lock (cond->__data.__lock);
+ lll_lock (cond->__data.__lock, pshared);
if (cond->__data.__total_seq > cond->__data.__wakeup_seq)
{
/* If there are still some waiters which have not been
woken up, this is an application bug. */
- lll_mutex_unlock (cond->__data.__lock);
+ lll_unlock (cond->__data.__lock, pshared);
return EBUSY;
}
@@ -45,7 +48,7 @@ __pthread_cond_destroy (cond)
pthread_cond_destroy needs to wait for them. */
unsigned int nwaiters = cond->__data.__nwaiters;
- if (nwaiters >= (1 << COND_CLOCK_BITS))
+ if (nwaiters >= (1 << COND_NWAITERS_SHIFT))
{
/* Wake everybody on the associated mutex in case there are
threads that have been requeued to it.
@@ -59,20 +62,21 @@ __pthread_cond_destroy (cond)
&& cond->__data.__mutex != (void *) ~0l)
{
pthread_mutex_t *mut = (pthread_mutex_t *) cond->__data.__mutex;
- lll_futex_wake (&mut->__data.__lock, INT_MAX);
+ lll_futex_wake (&mut->__data.__lock, INT_MAX,
+ PTHREAD_MUTEX_PSHARED (mut));
}
do
{
- lll_mutex_unlock (cond->__data.__lock);
+ lll_unlock (cond->__data.__lock, pshared);
- lll_futex_wait (&cond->__data.__nwaiters, nwaiters);
+ lll_futex_wait (&cond->__data.__nwaiters, nwaiters, pshared);
- lll_mutex_lock (cond->__data.__lock);
+ lll_lock (cond->__data.__lock, pshared);
nwaiters = cond->__data.__nwaiters;
}
- while (nwaiters >= (1 << COND_CLOCK_BITS));
+ while (nwaiters >= (1 << COND_NWAITERS_SHIFT));
}
return 0;
--- libc/nptl/pthread_cond_init.c 15 Apr 2005 22:19:22 -0000 1.11
+++ libc/nptl/pthread_cond_init.c 1 Aug 2007 04:14:08 -0000 1.13
@@ -28,11 +28,11 @@ __pthread_cond_init (cond, cond_attr)
{
struct pthread_condattr *icond_attr = (struct pthread_condattr *) cond_attr;
- cond->__data.__lock = LLL_MUTEX_LOCK_INITIALIZER;
+ cond->__data.__lock = LLL_LOCK_INITIALIZER;
cond->__data.__futex = 0;
cond->__data.__nwaiters = (icond_attr != NULL
- && ((icond_attr->value & (COND_CLOCK_BITS << 1))
- >> 1));
+ && ((icond_attr->value
+ & (COND_NWAITERS_SHIFT << 1)) >> 1));
cond->__data.__total_seq = 0;
cond->__data.__wakeup_seq = 0;
cond->__data.__woken_seq = 0;
--- libc/nptl/sysdeps/pthread/pthread_cond_signal.c 28 Oct 2006 05:07:42 -0000 1.4
+++ libc/nptl/sysdeps/pthread/pthread_cond_signal.c 11 Aug 2007 18:46:16 -0000 1.7
@@ -32,8 +32,11 @@ int
__pthread_cond_signal (cond)
pthread_cond_t *cond;
{
+ int pshared = (cond->__data.__mutex == (void *) ~0l)
+ ? LLL_SHARED : LLL_PRIVATE;
+
/* Make sure we are alone. */
- lll_mutex_lock (cond->__data.__lock);
+ lll_lock (cond->__data.__lock, pshared);
/* Are there any waiters to be woken? */
if (cond->__data.__total_seq > cond->__data.__wakeup_seq)
@@ -44,15 +47,15 @@ __pthread_cond_signal (cond)
/* Wake one. */
if (! __builtin_expect (lll_futex_wake_unlock (&cond->__data.__futex, 1,
- 1, &cond->__data.__lock),
- 0))
+ 1, &cond->__data.__lock,
+ pshared), 0))
return 0;
- lll_futex_wake (&cond->__data.__futex, 1);
+ lll_futex_wake (&cond->__data.__futex, 1, pshared);
}
/* We are done. */
- lll_mutex_unlock (cond->__data.__lock);
+ lll_unlock (cond->__data.__lock, pshared);
return 0;
}
--- libc/nptl/sysdeps/pthread/pthread_cond_timedwait.c 28 Oct 2006 05:08:17 -0000 1.5
+++ libc/nptl/sysdeps/pthread/pthread_cond_timedwait.c 13 Aug 2007 18:32:59 -0000 1.10
@@ -23,6 +23,7 @@
#include <lowlevellock.h>
#include <pthread.h>
#include <pthreadP.h>
+#include <kernel-features.h>
#include <shlib-compat.h>
@@ -53,21 +54,24 @@ __pthread_cond_timedwait (cond, mutex, a
if (abstime->tv_nsec < 0 || abstime->tv_nsec >= 1000000000)
return EINVAL;
+ int pshared = (cond->__data.__mutex == (void *) ~0l)
+ ? LLL_SHARED : LLL_PRIVATE;
+
/* Make sure we are along. */
- lll_mutex_lock (cond->__data.__lock);
+ lll_lock (cond->__data.__lock, pshared);
/* Now we can release the mutex. */
int err = __pthread_mutex_unlock_usercnt (mutex, 0);
if (err)
{
- lll_mutex_unlock (cond->__data.__lock);
+ lll_unlock (cond->__data.__lock, pshared);
return err;
}
/* We have one new user of the condvar. */
++cond->__data.__total_seq;
++cond->__data.__futex;
- cond->__data.__nwaiters += 1 << COND_CLOCK_BITS;
+ cond->__data.__nwaiters += 1 << COND_NWAITERS_SHIFT;
/* Remember the mutex we are using here. If there is already a
different address store this is a bad user bug. Do not store
@@ -100,7 +104,7 @@ __pthread_cond_timedwait (cond, mutex, a
int ret;
ret = INTERNAL_SYSCALL (clock_gettime, err, 2,
(cond->__data.__nwaiters
- & ((1 << COND_CLOCK_BITS) - 1)),
+ & ((1 << COND_NWAITERS_SHIFT) - 1)),
&rt);
# ifndef __ASSUME_POSIX_TIMERS
if (__builtin_expect (INTERNAL_SYSCALL_ERROR_P (ret, err), 0))
@@ -146,20 +150,20 @@ __pthread_cond_timedwait (cond, mutex, a
unsigned int futex_val = cond->__data.__futex;
/* Prepare to wait. Release the condvar futex. */
- lll_mutex_unlock (cond->__data.__lock);
+ lll_unlock (cond->__data.__lock, pshared);
/* Enable asynchronous cancellation. Required by the standard. */
cbuffer.oldtype = __pthread_enable_asynccancel ();
/* Wait until woken by signal or broadcast. */
err = lll_futex_timed_wait (&cond->__data.__futex,
- futex_val, &rt);
+ futex_val, &rt, pshared);
/* Disable asynchronous cancellation. */
__pthread_disable_asynccancel (cbuffer.oldtype);
/* We are going to look at shared data again, so get the lock. */
- lll_mutex_lock(cond->__data.__lock);
+ lll_lock (cond->__data.__lock, pshared);
/* If a broadcast happened, we are done. */
if (cbuffer.bc_seq != cond->__data.__broadcast_seq)
@@ -189,17 +193,17 @@ __pthread_cond_timedwait (cond, mutex, a
bc_out:
- cond->__data.__nwaiters -= 1 << COND_CLOCK_BITS;
+ cond->__data.__nwaiters -= 1 << COND_NWAITERS_SHIFT;
/* If pthread_cond_destroy was called on this variable already,
notify the pthread_cond_destroy caller all waiters have left
and it can be successfully destroyed. */
if (cond->__data.__total_seq == -1ULL
- && cond->__data.__nwaiters < (1 << COND_CLOCK_BITS))
- lll_futex_wake (&cond->__data.__nwaiters, 1);
+ && cond->__data.__nwaiters < (1 << COND_NWAITERS_SHIFT))
+ lll_futex_wake (&cond->__data.__nwaiters, 1, pshared);
/* We are done with the condvar. */
- lll_mutex_unlock (cond->__data.__lock);
+ lll_unlock (cond->__data.__lock, pshared);
/* The cancellation handling is back to normal, remove the handler. */
__pthread_cleanup_pop (&buffer, 0);
--- libc/nptl/sysdeps/pthread/pthread_cond_wait.c 28 Oct 2006 05:08:47 -0000 1.5
+++ libc/nptl/sysdeps/pthread/pthread_cond_wait.c 11 Aug 2007 18:44:27 -0000 1.9
@@ -43,9 +43,11 @@ __condvar_cleanup (void *arg)
struct _condvar_cleanup_buffer *cbuffer =
(struct _condvar_cleanup_buffer *) arg;
unsigned int destroying;
+ int pshared = (cbuffer->cond->__data.__mutex == (void *) ~0l)
+ ? LLL_SHARED : LLL_PRIVATE;
/* We are going to modify shared data. */
- lll_mutex_lock (cbuffer->cond->__data.__lock);
+ lll_lock (cbuffer->cond->__data.__lock, pshared);
if (cbuffer->bc_seq == cbuffer->cond->__data.__broadcast_seq)
{
@@ -62,25 +64,25 @@ __condvar_cleanup (void *arg)
++cbuffer->cond->__data.__woken_seq;
}
- cbuffer->cond->__data.__nwaiters -= 1 << COND_CLOCK_BITS;
+ cbuffer->cond->__data.__nwaiters -= 1 << COND_NWAITERS_SHIFT;
/* If pthread_cond_destroy was called on this variable already,
notify the pthread_cond_destroy caller all waiters have left
and it can be successfully destroyed. */
destroying = 0;
if (cbuffer->cond->__data.__total_seq == -1ULL
- && cbuffer->cond->__data.__nwaiters < (1 << COND_CLOCK_BITS))
+ && cbuffer->cond->__data.__nwaiters < (1 << COND_NWAITERS_SHIFT))
{
- lll_futex_wake (&cbuffer->cond->__data.__nwaiters, 1);
+ lll_futex_wake (&cbuffer->cond->__data.__nwaiters, 1, pshared);
destroying = 1;
}
/* We are done. */
- lll_mutex_unlock (cbuffer->cond->__data.__lock);
+ lll_unlock (cbuffer->cond->__data.__lock, pshared);
/* Wake everybody to make sure no condvar signal gets lost. */
if (! destroying)
- lll_futex_wake (&cbuffer->cond->__data.__futex, INT_MAX);
+ lll_futex_wake (&cbuffer->cond->__data.__futex, INT_MAX, pshared);
/* Get the mutex before returning unless asynchronous cancellation
is in effect. */
@@ -96,22 +98,24 @@ __pthread_cond_wait (cond, mutex)
struct _pthread_cleanup_buffer buffer;
struct _condvar_cleanup_buffer cbuffer;
int err;
+ int pshared = (cond->__data.__mutex == (void *) ~0l)
+ ? LLL_SHARED : LLL_PRIVATE;
/* Make sure we are along. */
- lll_mutex_lock (cond->__data.__lock);
+ lll_lock (cond->__data.__lock, pshared);
/* Now we can release the mutex. */
err = __pthread_mutex_unlock_usercnt (mutex, 0);
if (__builtin_expect (err, 0))
{
- lll_mutex_unlock (cond->__data.__lock);
+ lll_unlock (cond->__data.__lock, pshared);
return err;
}
/* We have one new user of the condvar. */
++cond->__data.__total_seq;
++cond->__data.__futex;
- cond->__data.__nwaiters += 1 << COND_CLOCK_BITS;
+ cond->__data.__nwaiters += 1 << COND_NWAITERS_SHIFT;
/* Remember the mutex we are using here. If there is already a
different address store this is a bad user bug. Do not store
@@ -140,19 +144,19 @@ __pthread_cond_wait (cond, mutex)
unsigned int futex_val = cond->__data.__futex;
/* Prepare to wait. Release the condvar futex. */
- lll_mutex_unlock (cond->__data.__lock);
+ lll_unlock (cond->__data.__lock, pshared);
/* Enable asynchronous cancellation. Required by the standard. */
cbuffer.oldtype = __pthread_enable_asynccancel ();
/* Wait until woken by signal or broadcast. */
- lll_futex_wait (&cond->__data.__futex, futex_val);
+ lll_futex_wait (&cond->__data.__futex, futex_val, pshared);
/* Disable asynchronous cancellation. */
__pthread_disable_asynccancel (cbuffer.oldtype);
/* We are going to look at shared data again, so get the lock. */
- lll_mutex_lock (cond->__data.__lock);
+ lll_lock (cond->__data.__lock, pshared);
/* If a broadcast happened, we are done. */
if (cbuffer.bc_seq != cond->__data.__broadcast_seq)
@@ -168,17 +172,17 @@ __pthread_cond_wait (cond, mutex)
bc_out:
- cond->__data.__nwaiters -= 1 << COND_CLOCK_BITS;
+ cond->__data.__nwaiters -= 1 << COND_NWAITERS_SHIFT;
/* If pthread_cond_destroy was called on this varaible already,
notify the pthread_cond_destroy caller all waiters have left
and it can be successfully destroyed. */
if (cond->__data.__total_seq == -1ULL
- && cond->__data.__nwaiters < (1 << COND_CLOCK_BITS))
- lll_futex_wake (&cond->__data.__nwaiters, 1);
+ && cond->__data.__nwaiters < (1 << COND_NWAITERS_SHIFT))
+ lll_futex_wake (&cond->__data.__nwaiters, 1, pshared);
/* We are done with the condvar. */
- lll_mutex_unlock (cond->__data.__lock);
+ lll_unlock (cond->__data.__lock, pshared);
/* The cancellation handling is back to normal, remove the handler. */
__pthread_cleanup_pop (&buffer, 0);
--- libc/nptl/pthread_condattr_getclock.c 2 Sep 2004 18:46:38 -0000 1.2
+++ libc/nptl/pthread_condattr_getclock.c 29 May 2007 16:15:40 -0000 1.3
@@ -26,6 +26,6 @@ pthread_condattr_getclock (attr, clock_i
clockid_t *clock_id;
{
*clock_id = (((((const struct pthread_condattr *) attr)->value) >> 1)
- & ((1 << COND_CLOCK_BITS) - 1));
+ & ((1 << COND_NWAITERS_SHIFT) - 1));
return 0;
}
--- libc/nptl/pthread_condattr_setclock.c 2 Sep 2004 18:47:34 -0000 1.3
+++ libc/nptl/pthread_condattr_setclock.c 29 May 2007 16:15:40 -0000 1.4
@@ -62,11 +62,12 @@ pthread_condattr_setclock (attr, clock_i
return EINVAL;
/* Make sure the value fits in the bits we reserved. */
- assert (clock_id < (1 << COND_CLOCK_BITS));
+ assert (clock_id < (1 << COND_NWAITERS_SHIFT));
int *valuep = &((struct pthread_condattr *) attr)->value;
- *valuep = (*valuep & ~(1 << (COND_CLOCK_BITS + 1)) & ~1) | (clock_id << 1);
+ *valuep = ((*valuep & ~(1 << (COND_NWAITERS_SHIFT + 1)) & ~1)
+ | (clock_id << 1));
return 0;
}
--- libc/nptl/pthread_create.c 5 Sep 2006 17:12:15 -0000 1.54
+++ libc/nptl/pthread_create.c 13 Aug 2007 18:32:59 -0000 1.58
@@ -27,6 +27,7 @@
#include <atomic.h>
#include <libc-internal.h>
#include <resolv.h>
+#include <kernel-features.h>
#include <shlib-compat.h>
@@ -63,7 +64,7 @@ __find_in_stack_list (pd)
list_t *entry;
struct pthread *result = NULL;
- lll_lock (stack_cache_lock);
+ lll_lock (stack_cache_lock, LLL_PRIVATE);
list_for_each (entry, &stack_used)
{
@@ -90,7 +91,7 @@ __find_in_stack_list (pd)
}
}
- lll_unlock (stack_cache_lock);
+ lll_unlock (stack_cache_lock, LLL_PRIVATE);
return result;
}
@@ -284,9 +285,9 @@ start_thread (void *arg)
int oldtype = CANCEL_ASYNC ();
/* Get the lock the parent locked to force synchronization. */
- lll_lock (pd->lock);
+ lll_lock (pd->lock, LLL_PRIVATE);
/* And give it up right away. */
- lll_unlock (pd->lock);
+ lll_unlock (pd->lock, LLL_PRIVATE);
CANCEL_RESET (oldtype);
}
@@ -370,7 +371,7 @@ start_thread (void *arg)
# endif
this->__list.__next = NULL;
- lll_robust_mutex_dead (this->__lock);
+ lll_robust_dead (this->__lock, /* XYZ */ LLL_SHARED);
}
while (robust != (void *) &pd->robust_head);
}
@@ -385,7 +386,7 @@ start_thread (void *arg)
/* Some other thread might call any of the setXid functions and expect
us to reply. In this case wait until we did that. */
do
- lll_futex_wait (&pd->setxid_futex, 0);
+ lll_futex_wait (&pd->setxid_futex, 0, LLL_PRIVATE);
while (pd->cancelhandling & SETXID_BITMASK);
/* Reset the value so that the stack can be reused. */
--- libc/nptl/pthread_getattr_np.c 7 Apr 2006 04:26:42 -0000 1.12
+++ libc/nptl/pthread_getattr_np.c 1 Aug 2007 04:19:49 -0000 1.15
@@ -39,11 +39,7 @@ pthread_getattr_np (thread_id, attr)
struct pthread_attr *iattr = (struct pthread_attr *) attr;
int ret = 0;
- /* We have to handle cancellation in the following code since we are
- locking another threads desriptor. */
- pthread_cleanup_push ((void (*) (void *)) lll_unlock_wake_cb, &thread->lock);
-
- lll_lock (thread->lock);
+ lll_lock (thread->lock, LLL_PRIVATE);
/* The thread library is responsible for keeping the values in the
thread desriptor up-to-date in case the user changes them. */
@@ -177,9 +173,7 @@ pthread_getattr_np (thread_id, attr)
}
}
- lll_unlock (thread->lock);
-
- pthread_cleanup_pop (0);
+ lll_unlock (thread->lock, LLL_PRIVATE);
return ret;
}
--- libc/nptl/pthread_getschedparam.c 27 Feb 2004 08:05:13 -0000 1.5
+++ libc/nptl/pthread_getschedparam.c 1 Aug 2007 04:10:41 -0000 1.7
@@ -38,11 +38,7 @@ __pthread_getschedparam (threadid, polic
int result = 0;
- /* We have to handle cancellation in the following code since we are
- locking another threads descriptor. */
- pthread_cleanup_push ((void (*) (void *)) lll_unlock_wake_cb, &pd->lock);
-
- lll_lock (pd->lock);
+ lll_lock (pd->lock, LLL_PRIVATE);
/* The library is responsible for maintaining the values at all
times. If the user uses a interface other than
@@ -72,9 +68,7 @@ __pthread_getschedparam (threadid, polic
memcpy (param, &pd->schedparam, sizeof (struct sched_param));
}
- lll_unlock (pd->lock);
-
- pthread_cleanup_pop (0);
+ lll_unlock (pd->lock, LLL_PRIVATE);
return result;
}
--- libc/nptl/pthread_mutex_init.c 14 Aug 2006 23:00:38 -0000 1.11
+++ libc/nptl/pthread_mutex_init.c 13 Aug 2007 18:32:59 -0000 1.13
@@ -20,6 +20,7 @@
#include <assert.h>
#include <errno.h>
#include <string.h>
+#include <kernel-features.h>
#include "pthreadP.h"
static const struct pthread_mutexattr default_attr =
@@ -120,6 +121,12 @@ __pthread_mutex_init (mutex, mutexattr)
break;
}
+ /* The kernel when waking robust mutexes on exit never uses
+ FUTEX_PRIVATE_FLAG FUTEX_WAKE. */
+ if ((imutexattr->mutexkind & (PTHREAD_MUTEXATTR_FLAG_PSHARED
+ | PTHREAD_MUTEXATTR_FLAG_ROBUST)) != 0)
+ mutex->__data.__kind |= PTHREAD_MUTEX_PSHARED_BIT;
+
/* Default values: mutex not used yet. */
// mutex->__count = 0; already done by memset
// mutex->__owner = 0; already done by memset
--- libc/nptl/pthread_mutex_lock.c 18 May 2007 00:53:33 -0000 1.16
+++ libc/nptl/pthread_mutex_lock.c 11 Aug 2007 18:46:53 -0000 1.20
@@ -27,9 +27,13 @@
#ifndef LLL_MUTEX_LOCK
-# define LLL_MUTEX_LOCK(mutex) lll_mutex_lock (mutex)
-# define LLL_MUTEX_TRYLOCK(mutex) lll_mutex_trylock (mutex)
-# define LLL_ROBUST_MUTEX_LOCK(mutex, id) lll_robust_mutex_lock (mutex, id)
+# define LLL_MUTEX_LOCK(mutex) \
+ lll_lock ((mutex)->__data.__lock, PTHREAD_MUTEX_PSHARED (mutex))
+# define LLL_MUTEX_TRYLOCK(mutex) \
+ lll_trylock ((mutex)->__data.__lock)
+# define LLL_ROBUST_MUTEX_LOCK(mutex, id) \
+ lll_robust_lock ((mutex)->__data.__lock, id, \
+ PTHREAD_ROBUST_MUTEX_PSHARED (mutex))
#endif
@@ -43,7 +47,8 @@ __pthread_mutex_lock (mutex)
pid_t id = THREAD_GETMEM (THREAD_SELF, tid);
int retval = 0;
- switch (__builtin_expect (mutex->__data.__kind, PTHREAD_MUTEX_TIMED_NP))
+ switch (__builtin_expect (PTHREAD_MUTEX_TYPE (mutex),
+ PTHREAD_MUTEX_TIMED_NP))
{
/* Recursive mutex. */
case PTHREAD_MUTEX_RECURSIVE_NP:
@@ -61,7 +66,7 @@ __pthread_mutex_lock (mutex)
}
/* We have to get the mutex. */
- LLL_MUTEX_LOCK (mutex->__data.__lock);
+ LLL_MUTEX_LOCK (mutex);
assert (mutex->__data.__owner == 0);
mutex->__data.__count = 1;
@@ -78,7 +83,7 @@ __pthread_mutex_lock (mutex)
case PTHREAD_MUTEX_TIMED_NP:
simple:
/* Normal mutex. */
- LLL_MUTEX_LOCK (mutex->__data.__lock);
+ LLL_MUTEX_LOCK (mutex);
assert (mutex->__data.__owner == 0);
break;
@@ -86,7 +91,7 @@ __pthread_mutex_lock (mutex)
if (! __is_smp)
goto simple;
- if (LLL_MUTEX_TRYLOCK (mutex->__data.__lock) != 0)
+ if (LLL_MUTEX_TRYLOCK (mutex) != 0)
{
int cnt = 0;
int max_cnt = MIN (MAX_ADAPTIVE_COUNT,
@@ -95,7 +100,7 @@ __pthread_mutex_lock (mutex)
{
if (cnt++ >= max_cnt)
{
- LLL_MUTEX_LOCK (mutex->__data.__lock);
+ LLL_MUTEX_LOCK (mutex);
break;
}
@@ -103,7 +108,7 @@ __pthread_mutex_lock (mutex)
BUSY_WAIT_NOP;
#endif
}
- while (LLL_MUTEX_TRYLOCK (mutex->__data.__lock) != 0);
+ while (LLL_MUTEX_TRYLOCK (mutex) != 0);
mutex->__data.__spins += (cnt - mutex->__data.__spins) / 8;
}
@@ -165,16 +170,15 @@ __pthread_mutex_lock (mutex)
/* Check whether we already hold the mutex. */
if (__builtin_expect ((oldval & FUTEX_TID_MASK) == id, 0))
{
- if (mutex->__data.__kind
- == PTHREAD_MUTEX_ROBUST_ERRORCHECK_NP)
+ int kind = PTHREAD_MUTEX_TYPE (mutex);
+ if (kind == PTHREAD_MUTEX_ROBUST_ERRORCHECK_NP)
{
THREAD_SETMEM (THREAD_SELF, robust_head.list_op_pending,
NULL);
return EDEADLK;
}
- if (mutex->__data.__kind
- == PTHREAD_MUTEX_ROBUST_RECURSIVE_NP)
+ if (kind == PTHREAD_MUTEX_ROBUST_RECURSIVE_NP)
{
THREAD_SETMEM (THREAD_SELF, robust_head.list_op_pending,
NULL);
@@ -190,14 +194,15 @@ __pthread_mutex_lock (mutex)
}
}
- oldval = LLL_ROBUST_MUTEX_LOCK (mutex->__data.__lock, id);
+ oldval = LLL_ROBUST_MUTEX_LOCK (mutex, id);
if (__builtin_expect (mutex->__data.__owner
== PTHREAD_MUTEX_NOTRECOVERABLE, 0))
{
/* This mutex is now not recoverable. */
mutex->__data.__count = 0;
- lll_mutex_unlock (mutex->__data.__lock);
+ lll_unlock (mutex->__data.__lock,
+ PTHREAD_ROBUST_MUTEX_PSHARED (mutex));
THREAD_SETMEM (THREAD_SELF, robust_head.list_op_pending, NULL);
return ENOTRECOVERABLE;
}
@@ -408,7 +413,8 @@ __pthread_mutex_lock (mutex)
break;
if (oldval != ceilval)
- lll_futex_wait (&mutex->__data.__lock, ceilval | 2);
+ lll_futex_wait (&mutex->__data.__lock, ceilval | 2,
+ PTHREAD_MUTEX_PSHARED (mutex));
}
while (atomic_compare_and_exchange_val_acq (&mutex->__data.__lock,
ceilval | 2, ceilval)
--- libc/nptl/pthread_mutex_setprioceiling.c 14 Aug 2006 23:05:35 -0000 1.3
+++ libc/nptl/pthread_mutex_setprioceiling.c 11 Aug 2007 18:45:57 -0000 1.5
@@ -47,12 +47,13 @@ pthread_mutex_setprioceiling (mutex, pri
/* Check whether we already hold the mutex. */
bool locked = false;
+ int kind = PTHREAD_MUTEX_TYPE (mutex);
if (mutex->__data.__owner == THREAD_GETMEM (THREAD_SELF, tid))
{
- if (mutex->__data.__kind == PTHREAD_MUTEX_PP_ERRORCHECK_NP)
+ if (kind == PTHREAD_MUTEX_PP_ERRORCHECK_NP)
return EDEADLK;
- if (mutex->__data.__kind == PTHREAD_MUTEX_PP_RECURSIVE_NP)
+ if (kind == PTHREAD_MUTEX_PP_RECURSIVE_NP)
locked = true;
}
@@ -80,7 +81,8 @@ pthread_mutex_setprioceiling (mutex, pri
break;
if (oldval != ceilval)
- lll_futex_wait (&mutex->__data.__lock, ceilval | 2);
+ lll_futex_wait (&mutex->__data.__lock, ceilval | 2,
+ PTHREAD_MUTEX_PSHARED (mutex));
}
while (atomic_compare_and_exchange_val_acq (&mutex->__data.__lock,
ceilval | 2, ceilval)
@@ -110,7 +112,8 @@ pthread_mutex_setprioceiling (mutex, pri
| (prioceiling << PTHREAD_MUTEX_PRIO_CEILING_SHIFT);
atomic_full_barrier ();
- lll_futex_wake (&mutex->__data.__lock, INT_MAX);
+ lll_futex_wake (&mutex->__data.__lock, INT_MAX,
+ PTHREAD_MUTEX_PSHARED (mutex));
return 0;
}
--- libc/nptl/pthread_mutex_timedlock.c 18 May 2007 00:53:33 -0000 1.12
+++ libc/nptl/pthread_mutex_timedlock.c 11 Aug 2007 18:42:56 -0000 1.16
@@ -37,7 +37,8 @@ pthread_mutex_timedlock (mutex, abstime)
/* We must not check ABSTIME here. If the thread does not block
abstime must not be checked for a valid value. */
- switch (__builtin_expect (mutex->__data.__kind, PTHREAD_MUTEX_TIMED_NP))
+ switch (__builtin_expect (PTHREAD_MUTEX_TYPE (mutex),
+ PTHREAD_MUTEX_TIMED_NP))
{
/* Recursive mutex. */
case PTHREAD_MUTEX_RECURSIVE_NP:
@@ -55,7 +56,8 @@ pthread_mutex_timedlock (mutex, abstime)
}
/* We have to get the mutex. */
- result = lll_mutex_timedlock (mutex->__data.__lock, abstime);
+ result = lll_timedlock (mutex->__data.__lock, abstime,
+ PTHREAD_MUTEX_PSHARED (mutex));
if (result != 0)
goto out;
@@ -75,14 +77,15 @@ pthread_mutex_timedlock (mutex, abstime)
case PTHREAD_MUTEX_TIMED_NP:
simple:
/* Normal mutex. */
- result = lll_mutex_timedlock (mutex->__data.__lock, abstime);
+ result = lll_timedlock (mutex->__data.__lock, abstime,
+ PTHREAD_MUTEX_PSHARED (mutex));
break;
case PTHREAD_MUTEX_ADAPTIVE_NP:
if (! __is_smp)
goto simple;
- if (lll_mutex_trylock (mutex->__data.__lock) != 0)
+ if (lll_trylock (mutex->__data.__lock) != 0)
{
int cnt = 0;
int max_cnt = MIN (MAX_ADAPTIVE_COUNT,
@@ -91,7 +94,8 @@ pthread_mutex_timedlock (mutex, abstime)
{
if (cnt++ >= max_cnt)
{
- result = lll_mutex_timedlock (mutex->__data.__lock, abstime);
+ result = lll_timedlock (mutex->__data.__lock, abstime,
+ PTHREAD_MUTEX_PSHARED (mutex));
break;
}
@@ -99,7 +103,7 @@ pthread_mutex_timedlock (mutex, abstime)
BUSY_WAIT_NOP;
#endif
}
- while (lll_mutex_trylock (mutex->__data.__lock) != 0);
+ while (lll_trylock (mutex->__data.__lock) != 0);
mutex->__data.__spins += (cnt - mutex->__data.__spins) / 8;
}
@@ -148,16 +152,15 @@ pthread_mutex_timedlock (mutex, abstime)
/* Check whether we already hold the mutex. */
if (__builtin_expect ((oldval & FUTEX_TID_MASK) == id, 0))
{
- if (mutex->__data.__kind
- == PTHREAD_MUTEX_ROBUST_ERRORCHECK_NP)
+ int kind = PTHREAD_MUTEX_TYPE (mutex);
+ if (kind == PTHREAD_MUTEX_ROBUST_ERRORCHECK_NP)
{
THREAD_SETMEM (THREAD_SELF, robust_head.list_op_pending,
NULL);
return EDEADLK;
}
- if (mutex->__data.__kind
- == PTHREAD_MUTEX_ROBUST_RECURSIVE_NP)
+ if (kind == PTHREAD_MUTEX_ROBUST_RECURSIVE_NP)
{
THREAD_SETMEM (THREAD_SELF, robust_head.list_op_pending,
NULL);
@@ -173,15 +176,16 @@ pthread_mutex_timedlock (mutex, abstime)
}
}
- result = lll_robust_mutex_timedlock (mutex->__data.__lock, abstime,
- id);
+ result = lll_robust_timedlock (mutex->__data.__lock, abstime, id,
+ PTHREAD_ROBUST_MUTEX_PSHARED (mutex));
if (__builtin_expect (mutex->__data.__owner
== PTHREAD_MUTEX_NOTRECOVERABLE, 0))
{
/* This mutex is now not recoverable. */
mutex->__data.__count = 0;
- lll_mutex_unlock (mutex->__data.__lock);
+ lll_unlock (mutex->__data.__lock,
+ PTHREAD_ROBUST_MUTEX_PSHARED (mutex));
THREAD_SETMEM (THREAD_SELF, robust_head.list_op_pending, NULL);
return ENOTRECOVERABLE;
}
@@ -441,7 +445,8 @@ pthread_mutex_timedlock (mutex, abstime)
}
lll_futex_timed_wait (&mutex->__data.__lock,
- ceilval | 2, &rt);
+ ceilval | 2, &rt,
+ PTHREAD_MUTEX_PSHARED (mutex));
}
}
while (atomic_compare_and_exchange_val_acq (&mutex->__data.__lock,
--- libc/nptl/pthread_mutex_trylock.c 18 May 2007 00:53:33 -0000 1.14
+++ libc/nptl/pthread_mutex_trylock.c 11 Aug 2007 18:47:09 -0000 1.17
@@ -31,7 +31,8 @@ __pthread_mutex_trylock (mutex)
int oldval;
pid_t id = THREAD_GETMEM (THREAD_SELF, tid);
- switch (__builtin_expect (mutex->__data.__kind, PTHREAD_MUTEX_TIMED_NP))
+ switch (__builtin_expect (PTHREAD_MUTEX_TYPE (mutex),
+ PTHREAD_MUTEX_TIMED_NP))
{
/* Recursive mutex. */
case PTHREAD_MUTEX_RECURSIVE_NP:
@@ -47,7 +48,7 @@ __pthread_mutex_trylock (mutex)
return 0;
}
- if (lll_mutex_trylock (mutex->__data.__lock) == 0)
+ if (lll_trylock (mutex->__data.__lock) == 0)
{
/* Record the ownership. */
mutex->__data.__owner = id;
@@ -67,7 +68,7 @@ __pthread_mutex_trylock (mutex)
case PTHREAD_MUTEX_TIMED_NP:
case PTHREAD_MUTEX_ADAPTIVE_NP:
/* Normal mutex. */
- if (lll_mutex_trylock (mutex->__data.__lock) != 0)
+ if (lll_trylock (mutex->__data.__lock) != 0)
break;
/* Record the ownership. */
@@ -121,16 +122,15 @@ __pthread_mutex_trylock (mutex)
/* Check whether we already hold the mutex. */
if (__builtin_expect ((oldval & FUTEX_TID_MASK) == id, 0))
{
- if (mutex->__data.__kind
- == PTHREAD_MUTEX_ROBUST_ERRORCHECK_NP)
+ int kind = PTHREAD_MUTEX_TYPE (mutex);
+ if (kind == PTHREAD_MUTEX_ROBUST_ERRORCHECK_NP)
{
THREAD_SETMEM (THREAD_SELF, robust_head.list_op_pending,
NULL);
return EDEADLK;
}
- if (mutex->__data.__kind
- == PTHREAD_MUTEX_ROBUST_RECURSIVE_NP)
+ if (kind == PTHREAD_MUTEX_ROBUST_RECURSIVE_NP)
{
THREAD_SETMEM (THREAD_SELF, robust_head.list_op_pending,
NULL);
@@ -146,7 +146,7 @@ __pthread_mutex_trylock (mutex)
}
}
- oldval = lll_robust_mutex_trylock (mutex->__data.__lock, id);
+ oldval = lll_robust_trylock (mutex->__data.__lock, id);
if (oldval != 0 && (oldval & FUTEX_OWNER_DIED) == 0)
{
THREAD_SETMEM (THREAD_SELF, robust_head.list_op_pending, NULL);
@@ -160,7 +160,8 @@ __pthread_mutex_trylock (mutex)
/* This mutex is now not recoverable. */
mutex->__data.__count = 0;
if (oldval == id)
- lll_mutex_unlock (mutex->__data.__lock);
+ lll_unlock (mutex->__data.__lock,
+ PTHREAD_ROBUST_MUTEX_PSHARED (mutex));
THREAD_SETMEM (THREAD_SELF, robust_head.list_op_pending, NULL);
return ENOTRECOVERABLE;
}
--- libc/nptl/pthread_mutex_unlock.c 14 Aug 2006 23:01:26 -0000 1.14
+++ libc/nptl/pthread_mutex_unlock.c 11 Aug 2007 18:43:23 -0000 1.18
@@ -31,7 +31,8 @@ __pthread_mutex_unlock_usercnt (mutex, d
{
int newowner = 0;
- switch (__builtin_expect (mutex->__data.__kind, PTHREAD_MUTEX_TIMED_NP))
+ switch (__builtin_expect (PTHREAD_MUTEX_TYPE (mutex),
+ PTHREAD_MUTEX_TIMED_NP))
{
case PTHREAD_MUTEX_RECURSIVE_NP:
/* Recursive mutex. */
@@ -46,7 +47,7 @@ __pthread_mutex_unlock_usercnt (mutex, d
case PTHREAD_MUTEX_ERRORCHECK_NP:
/* Error checking mutex. */
if (mutex->__data.__owner != THREAD_GETMEM (THREAD_SELF, tid)
- || ! lll_mutex_islocked (mutex->__data.__lock))
+ || ! lll_islocked (mutex->__data.__lock))
return EPERM;
/* FALLTHROUGH */
@@ -60,7 +61,7 @@ __pthread_mutex_unlock_usercnt (mutex, d
--mutex->__data.__nusers;
/* Unlock. */
- lll_mutex_unlock (mutex->__data.__lock);
+ lll_unlock (mutex->__data.__lock, PTHREAD_MUTEX_PSHARED (mutex));
break;
case PTHREAD_MUTEX_ROBUST_RECURSIVE_NP:
@@ -91,7 +92,7 @@ __pthread_mutex_unlock_usercnt (mutex, d
case PTHREAD_MUTEX_ROBUST_ADAPTIVE_NP:
if ((mutex->__data.__lock & FUTEX_TID_MASK)
!= THREAD_GETMEM (THREAD_SELF, tid)
- || ! lll_mutex_islocked (mutex->__data.__lock))
+ || ! lll_islocked (mutex->__data.__lock))
return EPERM;
/* If the previous owner died and the caller did not succeed in
@@ -114,7 +115,8 @@ __pthread_mutex_unlock_usercnt (mutex, d
--mutex->__data.__nusers;
/* Unlock. */
- lll_robust_mutex_unlock (mutex->__data.__lock);
+ lll_robust_unlock (mutex->__data.__lock,
+ PTHREAD_ROBUST_MUTEX_PSHARED (mutex));
THREAD_SETMEM (THREAD_SELF, robust_head.list_op_pending, NULL);
break;
@@ -160,7 +162,7 @@ __pthread_mutex_unlock_usercnt (mutex, d
case PTHREAD_MUTEX_PI_ROBUST_ADAPTIVE_NP:
if ((mutex->__data.__lock & FUTEX_TID_MASK)
!= THREAD_GETMEM (THREAD_SELF, tid)
- || ! lll_mutex_islocked (mutex->__data.__lock))
+ || ! lll_islocked (mutex->__data.__lock))
return EPERM;
/* If the previous owner died and the caller did not succeed in
@@ -240,7 +242,8 @@ __pthread_mutex_unlock_usercnt (mutex, d
newval, oldval));
if ((oldval & ~PTHREAD_MUTEX_PRIO_CEILING_MASK) > 1)
- lll_futex_wake (&mutex->__data.__lock, 1);
+ lll_futex_wake (&mutex->__data.__lock, 1,
+ PTHREAD_MUTEX_PSHARED (mutex));
int oldprio = newval >> PTHREAD_MUTEX_PRIO_CEILING_SHIFT;
return __pthread_tpp_change_priority (oldprio, -1);
--- libc/nptl/sysdeps/pthread/pthread_once.c 28 Oct 2006 05:09:12 -0000 1.1
+++ libc/nptl/sysdeps/pthread/pthread_once.c 1 Aug 2007 04:16:08 -0000 1.2
@@ -22,7 +22,7 @@
-static lll_lock_t once_lock = LLL_LOCK_INITIALIZER;
+static int once_lock = LLL_LOCK_INITIALIZER;
int
@@ -35,7 +35,7 @@ __pthread_once (once_control, init_routi
object. */
if (*once_control == PTHREAD_ONCE_INIT)
{
- lll_lock (once_lock);
+ lll_lock (once_lock, LLL_PRIVATE);
/* XXX This implementation is not complete. It doesn't take
cancelation and fork into account. */
@@ -46,7 +46,7 @@ __pthread_once (once_control, init_routi
*once_control = !PTHREAD_ONCE_INIT;
}
- lll_unlock (once_lock);
+ lll_unlock (once_lock, LLL_PRIVATE);
}
return 0;
--- libc/nptl/pthread_rwlock_init.c 26 Nov 2002 22:49:43 -0000 1.1.1.1
+++ libc/nptl/pthread_rwlock_init.c 27 May 2007 17:58:50 -0000 1.2
@@ -18,6 +18,7 @@
02111-1307 USA. */
#include "pthreadP.h"
+#include <kernel-features.h>
static const struct pthread_rwlockattr default_attr =
@@ -37,14 +38,44 @@ __pthread_rwlock_init (rwlock, attr)
iattr = ((const struct pthread_rwlockattr *) attr) ?: &default_attr;
rwlock->__data.__lock = 0;
- rwlock->__data.__flags
- = iattr->lockkind == PTHREAD_RWLOCK_PREFER_WRITER_NONRECURSIVE_NP;
rwlock->__data.__nr_readers = 0;
- rwlock->__data.__writer = 0;
rwlock->__data.__readers_wakeup = 0;
rwlock->__data.__writer_wakeup = 0;
rwlock->__data.__nr_readers_queued = 0;
rwlock->__data.__nr_writers_queued = 0;
+ rwlock->__data.__writer = 0;
+
+ rwlock->__data.__flags
+ = iattr->lockkind == PTHREAD_RWLOCK_PREFER_WRITER_NONRECURSIVE_NP;
+
+ /* The __SHARED field is computed to minimize the work that needs to
+ be done while handling the futex. There are two inputs: the
+ availability of private futexes and whether the rwlock is shared
+ or private. Unfortunately the value of a private rwlock is
+ fixed: it must be zero. The PRIVATE_FUTEX flag has the value
+ 0x80 in case private futexes are available and zero otherwise.
+ This leads to the following table:
+
+ | pshared | result
+ | shared private | shared private |
+ ------------+-----------------+-----------------+
+ !avail 0 | 0 0 | 0 0 |
+ avail 0x80 | 0x80 0 | 0 0x80 |
+
+ If the pshared value is in locking functions XORed with avail
+ we get the expected result. */
+#ifdef __ASSUME_PRIVATE_FUTEX
+ rwlock->__data.__shared = (iattr->pshared == PTHREAD_PROCESS_PRIVATE
+ ? 0 : FUTEX_PRIVATE_FLAG);
+#else
+ rwlock->__data.__shared = (iattr->pshared == PTHREAD_PROCESS_PRIVATE
+ ? 0
+ : THREAD_GETMEM (THREAD_SELF,
+ header.private_futex));
+#endif
+
+ rwlock->__data.__pad1 = 0;
+ rwlock->__data.__pad2 = 0;
return 0;
}
--- libc/nptl/sysdeps/pthread/pthread_rwlock_rdlock.c 28 Oct 2006 05:09:40 -0000 1.1
+++ libc/nptl/sysdeps/pthread/pthread_rwlock_rdlock.c 1 Aug 2007 04:11:44 -0000 1.4
@@ -32,7 +32,7 @@ __pthread_rwlock_rdlock (rwlock)
int result = 0;
/* Make sure we are along. */
- lll_mutex_lock (rwlock->__data.__lock);
+ lll_lock (rwlock->__data.__lock, rwlock->__data.__shared);
while (1)
{
@@ -74,19 +74,20 @@ __pthread_rwlock_rdlock (rwlock)
int waitval = rwlock->__data.__readers_wakeup;
/* Free the lock. */
- lll_mutex_unlock (rwlock->__data.__lock);
+ lll_unlock (rwlock->__data.__lock, rwlock->__data.__shared);
/* Wait for the writer to finish. */
- lll_futex_wait (&rwlock->__data.__readers_wakeup, waitval);
+ lll_futex_wait (&rwlock->__data.__readers_wakeup, waitval,
+ rwlock->__data.__shared);
/* Get the lock. */
- lll_mutex_lock (rwlock->__data.__lock);
+ lll_lock (rwlock->__data.__lock, rwlock->__data.__shared);
--rwlock->__data.__nr_readers_queued;
}
/* We are done, free the lock. */
- lll_mutex_unlock (rwlock->__data.__lock);
+ lll_unlock (rwlock->__data.__lock, rwlock->__data.__shared);
return result;
}
--- libc/nptl/sysdeps/pthread/pthread_rwlock_timedrdlock.c 28 Oct 2006 05:10:11 -0000 1.1
+++ libc/nptl/sysdeps/pthread/pthread_rwlock_timedrdlock.c 1 Aug 2007 04:16:53 -0000 1.4
@@ -33,7 +33,7 @@ pthread_rwlock_timedrdlock (rwlock, abst
int result = 0;
/* Make sure we are along. */
- lll_mutex_lock(rwlock->__data.__lock);
+ lll_lock(rwlock->__data.__lock, rwlock->__data.__shared);
while (1)
{
@@ -110,14 +110,14 @@ pthread_rwlock_timedrdlock (rwlock, abst
int waitval = rwlock->__data.__readers_wakeup;
/* Free the lock. */
- lll_mutex_unlock (rwlock->__data.__lock);
+ lll_unlock (rwlock->__data.__lock, rwlock->__data.__shared);
/* Wait for the writer to finish. */
err = lll_futex_timed_wait (&rwlock->__data.__readers_wakeup,
- waitval, &rt);
+ waitval, &rt, rwlock->__data.__shared);
/* Get the lock. */
- lll_mutex_lock (rwlock->__data.__lock);
+ lll_lock (rwlock->__data.__lock, rwlock->__data.__shared);
--rwlock->__data.__nr_readers_queued;
@@ -131,7 +131,7 @@ pthread_rwlock_timedrdlock (rwlock, abst
}
/* We are done, free the lock. */
- lll_mutex_unlock (rwlock->__data.__lock);
+ lll_unlock (rwlock->__data.__lock, rwlock->__data.__shared);
return result;
}
--- libc/nptl/sysdeps/pthread/pthread_rwlock_timedwrlock.c 28 Oct 2006 05:10:46 -0000 1.1
+++ libc/nptl/sysdeps/pthread/pthread_rwlock_timedwrlock.c 1 Aug 2007 04:12:56 -0000 1.3
@@ -33,7 +33,7 @@ pthread_rwlock_timedwrlock (rwlock, abst
int result = 0;
/* Make sure we are along. */
- lll_mutex_lock (rwlock->__data.__lock);
+ lll_lock (rwlock->__data.__lock, rwlock->__data.__shared);
while (1)
{
@@ -100,14 +100,14 @@ pthread_rwlock_timedwrlock (rwlock, abst
int waitval = rwlock->__data.__writer_wakeup;
/* Free the lock. */
- lll_mutex_unlock (rwlock->__data.__lock);
+ lll_unlock (rwlock->__data.__lock, rwlock->__data.__shared);
/* Wait for the writer or reader(s) to finish. */
err = lll_futex_timed_wait (&rwlock->__data.__writer_wakeup,
- waitval, &rt);
+ waitval, &rt, rwlock->__data.__shared);
/* Get the lock. */
- lll_mutex_lock (rwlock->__data.__lock);
+ lll_lock (rwlock->__data.__lock, rwlock->__data.__shared);
/* To start over again, remove the thread from the writer list. */
--rwlock->__data.__nr_writers_queued;
@@ -121,7 +121,7 @@ pthread_rwlock_timedwrlock (rwlock, abst
}
/* We are done, free the lock. */
- lll_mutex_unlock (rwlock->__data.__lock);
+ lll_unlock (rwlock->__data.__lock, rwlock->__data.__shared);
return result;
}
--- libc/nptl/pthread_rwlock_tryrdlock.c 26 Nov 2002 22:49:44 -0000 1.1.1.1
+++ libc/nptl/pthread_rwlock_tryrdlock.c 1 Aug 2007 04:10:05 -0000 1.3
@@ -28,7 +28,7 @@ __pthread_rwlock_tryrdlock (rwlock)
{
int result = EBUSY;
- lll_mutex_lock (rwlock->__data.__lock);
+ lll_lock (rwlock->__data.__lock, rwlock->__data.__shared);
if (rwlock->__data.__writer == 0
&& (rwlock->__data.__nr_writers_queued == 0
@@ -43,7 +43,7 @@ __pthread_rwlock_tryrdlock (rwlock)
result = 0;
}
- lll_mutex_unlock (rwlock->__data.__lock);
+ lll_unlock (rwlock->__data.__lock, rwlock->__data.__shared);
return result;
}
--- libc/nptl/pthread_rwlock_trywrlock.c 31 Aug 2006 17:13:40 -0000 1.5
+++ libc/nptl/pthread_rwlock_trywrlock.c 1 Aug 2007 04:10:21 -0000 1.6
@@ -28,7 +28,7 @@ __pthread_rwlock_trywrlock (rwlock)
{
int result = EBUSY;
- lll_mutex_lock (rwlock->__data.__lock);
+ lll_lock (rwlock->__data.__lock, rwlock->__data.__shared);
if (rwlock->__data.__writer == 0 && rwlock->__data.__nr_readers == 0)
{
@@ -36,7 +36,7 @@ __pthread_rwlock_trywrlock (rwlock)
result = 0;
}
- lll_mutex_unlock (rwlock->__data.__lock);
+ lll_unlock (rwlock->__data.__lock, rwlock->__data.__shared);
return result;
}
--- libc/nptl/sysdeps/pthread/pthread_rwlock_unlock.c 28 Oct 2006 05:11:52 -0000 1.1
+++ libc/nptl/sysdeps/pthread/pthread_rwlock_unlock.c 1 Aug 2007 04:12:38 -0000 1.3
@@ -27,7 +27,7 @@
int
__pthread_rwlock_unlock (pthread_rwlock_t *rwlock)
{
- lll_mutex_lock (rwlock->__data.__lock);
+ lll_lock (rwlock->__data.__lock, rwlock->__data.__shared);
if (rwlock->__data.__writer)
rwlock->__data.__writer = 0;
else
@@ -37,19 +37,21 @@ __pthread_rwlock_unlock (pthread_rwlock_
if (rwlock->__data.__nr_writers_queued)
{
++rwlock->__data.__writer_wakeup;
- lll_mutex_unlock (rwlock->__data.__lock);
- lll_futex_wake (&rwlock->__data.__writer_wakeup, 1);
+ lll_unlock (rwlock->__data.__lock, rwlock->__data.__shared);
+ lll_futex_wake (&rwlock->__data.__writer_wakeup, 1,
+ rwlock->__data.__shared);
return 0;
}
else if (rwlock->__data.__nr_readers_queued)
{
++rwlock->__data.__readers_wakeup;
- lll_mutex_unlock (rwlock->__data.__lock);
- lll_futex_wake (&rwlock->__data.__readers_wakeup, INT_MAX);
+ lll_unlock (rwlock->__data.__lock, rwlock->__data.__shared);
+ lll_futex_wake (&rwlock->__data.__readers_wakeup, INT_MAX,
+ rwlock->__data.__shared);
return 0;
}
}
- lll_mutex_unlock (rwlock->__data.__lock);
+ lll_unlock (rwlock->__data.__lock, rwlock->__data.__shared);
return 0;
}
--- libc/nptl/sysdeps/pthread/pthread_rwlock_wrlock.c 28 Oct 2006 05:11:15 -0000 1.1
+++ libc/nptl/sysdeps/pthread/pthread_rwlock_wrlock.c 1 Aug 2007 04:12:23 -0000 1.3
@@ -32,7 +32,7 @@ __pthread_rwlock_wrlock (rwlock)
int result = 0;
/* Make sure we are along. */
- lll_mutex_lock (rwlock->__data.__lock);
+ lll_lock (rwlock->__data.__lock, rwlock->__data.__shared);
while (1)
{
@@ -65,20 +65,21 @@ __pthread_rwlock_wrlock (rwlock)
int waitval = rwlock->__data.__writer_wakeup;
/* Free the lock. */
- lll_mutex_unlock (rwlock->__data.__lock);
+ lll_unlock (rwlock->__data.__lock, rwlock->__data.__shared);
/* Wait for the writer or reader(s) to finish. */
- lll_futex_wait (&rwlock->__data.__writer_wakeup, waitval);
+ lll_futex_wait (&rwlock->__data.__writer_wakeup, waitval,
+ rwlock->__data.__shared);
/* Get the lock. */
- lll_mutex_lock (rwlock->__data.__lock);
+ lll_lock (rwlock->__data.__lock, rwlock->__data.__shared);
/* To start over again, remove the thread from the writer list. */
--rwlock->__data.__nr_writers_queued;
}
/* We are done, free the lock. */
- lll_mutex_unlock (rwlock->__data.__lock);
+ lll_unlock (rwlock->__data.__lock, rwlock->__data.__shared);
return result;
}
--- libc/nptl/pthread_setschedparam.c 14 Aug 2006 23:03:23 -0000 1.5
+++ libc/nptl/pthread_setschedparam.c 1 Aug 2007 04:14:51 -0000 1.7
@@ -39,11 +39,7 @@ __pthread_setschedparam (threadid, polic
int result = 0;
- /* We have to handle cancellation in the following code since we are
- locking another threads desriptor. */
- pthread_cleanup_push ((void (*) (void *)) lll_unlock_wake_cb, &pd->lock);
-
- lll_lock (pd->lock);
+ lll_lock (pd->lock, LLL_PRIVATE);
struct sched_param p;
const struct sched_param *orig_param = param;
@@ -71,9 +67,7 @@ __pthread_setschedparam (threadid, polic
pd->flags |= ATTR_FLAG_SCHED_SET | ATTR_FLAG_POLICY_SET;
}
- lll_unlock (pd->lock);
-
- pthread_cleanup_pop (0);
+ lll_unlock (pd->lock, LLL_PRIVATE);
return result;
}
--- libc/nptl/pthread_setschedprio.c 14 Aug 2006 23:03:23 -0000 1.2
+++ libc/nptl/pthread_setschedprio.c 1 Aug 2007 04:17:58 -0000 1.4
@@ -41,11 +41,7 @@ pthread_setschedprio (threadid, prio)
struct sched_param param;
param.sched_priority = prio;
- /* We have to handle cancellation in the following code since we are
- locking another threads desriptor. */
- pthread_cleanup_push ((void (*) (void *)) lll_unlock_wake_cb, &pd->lock);
-
- lll_lock (pd->lock);
+ lll_lock (pd->lock, LLL_PRIVATE);
/* If the thread should have higher priority because of some
PTHREAD_PRIO_PROTECT mutexes it holds, adjust the priority. */
@@ -64,9 +60,7 @@ pthread_setschedprio (threadid, prio)
pd->flags |= ATTR_FLAG_SCHED_SET;
}
- lll_unlock (pd->lock);
-
- pthread_cleanup_pop (0);
+ lll_unlock (pd->lock, LLL_PRIVATE);
return result;
}
--- libc/nptl/sem_close.c 17 May 2003 20:49:02 -0000 1.2
+++ libc/nptl/sem_close.c 1 Aug 2007 04:15:35 -0000 1.3
@@ -47,7 +47,7 @@ sem_close (sem)
int result = 0;
/* Get the lock. */
- lll_lock (__sem_mappings_lock);
+ lll_lock (__sem_mappings_lock, LLL_PRIVATE);
/* Locate the entry for the mapping the caller provided. */
rec = NULL;
@@ -75,7 +75,7 @@ sem_close (sem)
}
/* Release the lock. */
- lll_unlock (__sem_mappings_lock);
+ lll_unlock (__sem_mappings_lock, LLL_PRIVATE);
return result;
}
--- libc/nptl/sem_getvalue.c 16 Dec 2002 22:54:51 -0000 1.2
+++ libc/nptl/sem_getvalue.c 26 May 2007 01:29:25 -0000 1.3
@@ -27,11 +27,11 @@ __new_sem_getvalue (sem, sval)
sem_t *sem;
int *sval;
{
- struct sem *isem = (struct sem *) sem;
+ struct new_sem *isem = (struct new_sem *) sem;
/* XXX Check for valid SEM parameter. */
- *sval = isem->count;
+ *sval = isem->value;
return 0;
}
--- libc/nptl/sem_init.c 16 Dec 2002 22:56:03 -0000 1.2
+++ libc/nptl/sem_init.c 26 May 2007 01:29:04 -0000 1.3
@@ -22,6 +22,7 @@
#include <lowlevellock.h>
#include <shlib-compat.h>
#include "semaphoreP.h"
+#include <kernel-features.h>
int
@@ -38,18 +39,50 @@ __new_sem_init (sem, pshared, value)
}
/* Map to the internal type. */
- struct sem *isem = (struct sem *) sem;
+ struct new_sem *isem = (struct new_sem *) sem;
- /* Use the value the user provided. */
- isem->count = value;
+ /* Use the values the user provided. */
+ isem->value = value;
+#ifdef __ASSUME_PRIVATE_FUTEX
+ isem->private = pshared ? 0 : FUTEX_PRIVATE_FLAG;
+#else
+ isem->private = pshared ? 0 : THREAD_GETMEM (THREAD_SELF,
+ header.private_futex);
+#endif
- /* We can completely ignore the PSHARED parameter since inter-process
- use needs no special preparation. */
+ isem->nwaiters = 0;
return 0;
}
versioned_symbol (libpthread, __new_sem_init, sem_init, GLIBC_2_1);
+
+
+
#if SHLIB_COMPAT(libpthread, GLIBC_2_0, GLIBC_2_1)
-strong_alias (__new_sem_init, __old_sem_init)
+int
+attribute_compat_text_section
+__old_sem_init (sem, pshared, value)
+ sem_t *sem;
+ int pshared;
+ unsigned int value;
+{
+ /* Parameter sanity check. */
+ if (__builtin_expect (value > SEM_VALUE_MAX, 0))
+ {
+ __set_errno (EINVAL);
+ return -1;
+ }
+
+ /* Map to the internal type. */
+ struct old_sem *isem = (struct old_sem *) sem;
+
+ /* Use the value the user provided. */
+ isem->value = value;
+
+ /* We cannot store the PSHARED attribute. So we always use the
+ operations needed for shared semaphores. */
+
+ return 0;
+}
compat_symbol (libpthread, __old_sem_init, sem_init, GLIBC_2_0);
#endif
--- libc/nptl/sem_open.c 11 May 2006 17:09:43 -0000 1.11
+++ libc/nptl/sem_open.c 1 Aug 2007 04:07:13 -0000 1.13
@@ -147,7 +147,7 @@ __sem_search (const void *a, const void
void *__sem_mappings attribute_hidden;
/* Lock to protect the search tree. */
-lll_lock_t __sem_mappings_lock attribute_hidden = LLL_LOCK_INITIALIZER;
+int __sem_mappings_lock attribute_hidden = LLL_LOCK_INITIALIZER;
/* Search for existing mapping and if possible add the one provided. */
@@ -161,7 +161,7 @@ check_add_mapping (const char *name, siz
if (__fxstat64 (_STAT_VER, fd, &st) == 0)
{
/* Get the lock. */
- lll_lock (__sem_mappings_lock);
+ lll_lock (__sem_mappings_lock, LLL_PRIVATE);
/* Search for an existing mapping given the information we have. */
struct inuse_sem *fake;
@@ -210,7 +210,7 @@ check_add_mapping (const char *name, siz
}
/* Release the lock. */
- lll_unlock (__sem_mappings_lock);
+ lll_unlock (__sem_mappings_lock, LLL_PRIVATE);
}
if (result != existing && existing != SEM_FAILED && existing != MAP_FAILED)
@@ -304,12 +304,14 @@ sem_open (const char *name, int oflag, .
/* Create the initial file content. */
sem_t initsem;
- struct sem *iinitsem = (struct sem *) &initsem;
- iinitsem->count = value;
+ struct new_sem *iinitsem = (struct new_sem *) &initsem;
+ iinitsem->value = value;
+ iinitsem->private = 0;
+ iinitsem->nwaiters = 0;
/* Initialize the remaining bytes as well. */
- memset ((char *) &initsem + sizeof (struct sem), '\0',
- sizeof (sem_t) - sizeof (struct sem));
+ memset ((char *) &initsem + sizeof (struct new_sem), '\0',
+ sizeof (sem_t) - sizeof (struct new_sem));
tmpfname = (char *) alloca (mountpoint.dirlen + 6 + 1);
char *xxxxxx = __mempcpy (tmpfname, mountpoint.dir, mountpoint.dirlen);
--- libc/nptl/semaphoreP.h 11 May 2006 17:08:37 -0000 1.3
+++ libc/nptl/semaphoreP.h 1 Aug 2007 04:15:47 -0000 1.5
@@ -48,7 +48,7 @@ extern pthread_once_t __namedsem_once at
extern void *__sem_mappings attribute_hidden;
/* Lock to protect the search tree. */
-extern lll_lock_t __sem_mappings_lock attribute_hidden;
+extern int __sem_mappings_lock attribute_hidden;
/* Initializer for mountpoint. */
@@ -60,8 +60,10 @@ extern int __sem_search (const void *a,
/* Prototypes of functions with multiple interfaces. */
extern int __new_sem_init (sem_t *sem, int pshared, unsigned int value);
+extern int __old_sem_init (sem_t *sem, int pshared, unsigned int value);
extern int __new_sem_destroy (sem_t *sem);
extern int __new_sem_post (sem_t *sem);
extern int __new_sem_wait (sem_t *sem);
+extern int __old_sem_wait (sem_t *sem);
extern int __new_sem_trywait (sem_t *sem);
extern int __new_sem_getvalue (sem_t *sem, int *sval);
--- libc/nptl/tpp.c 14 Aug 2006 23:02:29 -0000 1.1
+++ libc/nptl/tpp.c 1 Aug 2007 04:47:15 -0000 1.2
@@ -93,7 +93,7 @@ __pthread_tpp_change_priority (int previ
if (priomax == newpriomax)
return 0;
- lll_lock (self->lock);
+ lll_lock (self->lock, LLL_PRIVATE);
tpp->priomax = newpriomax;
@@ -129,7 +129,7 @@ __pthread_tpp_change_priority (int previ
}
}
- lll_unlock (self->lock);
+ lll_unlock (self->lock, LLL_PRIVATE);
return result;
}
@@ -144,7 +144,7 @@ __pthread_current_priority (void)
int result = 0;
- lll_lock (self->lock);
+ lll_lock (self->lock, LLL_PRIVATE);
if ((self->flags & ATTR_FLAG_SCHED_SET) == 0)
{
@@ -166,7 +166,7 @@ __pthread_current_priority (void)
if (result != -1)
result = self->schedparam.sched_priority;
- lll_unlock (self->lock);
+ lll_unlock (self->lock, LLL_PRIVATE);
return result;
}
--- libc/nptl/tst-sem11.c 1 Jan 1970 00:00:00 -0000
+++ libc/nptl/tst-sem11.c 26 May 2007 01:23:04 -0000 1.1
@@ -0,0 +1,76 @@
+#include <semaphore.h>
+#include <stdio.h>
+#include <unistd.h>
+#include <pthread.h>
+#include <internaltypes.h>
+
+#ifndef SEM_WAIT
+# define SEM_WAIT(s) sem_wait (s)
+#endif
+
+static void *
+tf (void *arg)
+{
+#ifdef PREPARE
+ PREPARE
+#endif
+ SEM_WAIT (arg);
+ return NULL;
+}
+
+int
+main (void)
+{
+ int tries = 5;
+ pthread_t th;
+ sem_t s;
+ again:
+ if (sem_init (&s, 0, 0) != 0)
+ {
+ puts ("sem_init failed");
+ return 1;
+ }
+
+ struct new_sem *is = (struct new_sem *) &s;
+
+ if (is->nwaiters != 0)
+ {
+ puts ("nwaiters not initialized");
+ return 1;
+ }
+
+ if (pthread_create (&th, NULL, tf, &s) != 0)
+ {
+ puts ("pthread_create failed");
+ return 1;
+ }
+
+ sleep (1);
+
+ if (pthread_cancel (th) != 0)
+ {
+ puts ("pthread_cancel failed");
+ return 1;
+ }
+
+ void *r;
+ if (pthread_join (th, &r) != 0)
+ {
+ puts ("pthread_join failed");
+ return 1;
+ }
+ if (r != PTHREAD_CANCELED && --tries > 0)
+ {
+ /* Maybe we get the scheduling right the next time. */
+ sem_destroy (&s);
+ goto again;
+ }
+
+ if (is->nwaiters != 0)
+ {
+ puts ("nwaiters not reset");
+ return 1;
+ }
+
+ return 0;
+}
--- libc/nptl/tst-sem12.c 1 Jan 1970 00:00:00 -0000
+++ libc/nptl/tst-sem12.c 26 May 2007 01:23:04 -0000 1.1
@@ -0,0 +1,14 @@
+#include <time.h>
+#include <sys/time.h>
+
+
+#define PREPARE \
+ struct timespec ts; \
+ struct timeval tv; \
+ gettimeofday (&tv, NULL); \
+ TIMEVAL_TO_TIMESPEC (&tv, &ts); \
+ ts.tv_sec += 60;
+
+#define SEM_WAIT(s) sem_timedwait (s, &ts)
+
+#include "tst-sem11.c"
--- libc/nptl/tst-typesizes.c 27 Dec 2005 00:38:01 -0000 1.1
+++ libc/nptl/tst-typesizes.c 26 May 2007 01:29:58 -0000 1.2
@@ -59,7 +59,8 @@ do_test (void)
TEST_TYPE2 (pthread_rwlockattr_t, struct pthread_rwlockattr);
TEST_TYPE2 (pthread_barrier_t, struct pthread_barrier);
TEST_TYPE2 (pthread_barrierattr_t, struct pthread_barrierattr);
- TEST_TYPE2 (sem_t, struct sem);
+ TEST_TYPE2 (sem_t, struct new_sem);
+ TEST_TYPE2 (sem_t, struct old_sem);
return result;
}
--- libc/nptl/sysdeps/i386/tcb-offsets.sym 18 Dec 2005 06:56:18 -0000 1.8
+++ libc/nptl/sysdeps/i386/tcb-offsets.sym 23 May 2007 20:49:58 -0000 1.9
@@ -12,3 +12,6 @@ CLEANUP offsetof (struct pthread, clea
CLEANUP_PREV offsetof (struct _pthread_cleanup_buffer, __prev)
MUTEX_FUTEX offsetof (pthread_mutex_t, __data.__lock)
POINTER_GUARD offsetof (tcbhead_t, pointer_guard)
+#ifndef __ASSUME_PRIVATE_FUTEX
+PRIVATE_FUTEX offsetof (tcbhead_t, private_futex)
+#endif
--- libc/nptl/sysdeps/i386/tls.h 19 May 2007 07:06:24 -0000 1.34
+++ libc/nptl/sysdeps/i386/tls.h 28 Jul 2007 19:30:20 -0000 1.41
@@ -52,6 +52,9 @@ typedef struct
uintptr_t stack_guard;
uintptr_t pointer_guard;
int gscope_flag;
+#ifndef __ASSUME_PRIVATE_FUTEX
+ int private_futex;
+#endif
} tcbhead_t;
# define TLS_MULTIPLE_THREADS_IN_TCB 1
@@ -76,6 +79,7 @@ typedef struct
#ifndef __ASSEMBLER__
/* Get system call information. */
# include <sysdep.h>
+# include <kernel-features.h>
/* The old way: using LDT. */
@@ -447,7 +451,7 @@ union user_desc_init
: "i" (offsetof (struct pthread, header.gscope_flag)), \
"0" (THREAD_GSCOPE_FLAG_UNUSED)); \
if (__res == THREAD_GSCOPE_FLAG_WAIT) \
- lll_futex_wake (&THREAD_SELF->header.gscope_flag, 1); \
+ lll_futex_wake (&THREAD_SELF->header.gscope_flag, 1, LLL_PRIVATE); \
} \
while (0)
#define THREAD_GSCOPE_SET_FLAG() \
--- libc/nptl/sysdeps/powerpc/tcb-offsets.sym 20 Dec 2005 06:44:05 -0000 1.7
+++ libc/nptl/sysdeps/powerpc/tcb-offsets.sym 26 May 2007 20:13:18 -0000 1.9
@@ -15,3 +15,6 @@ MULTIPLE_THREADS_OFFSET thread_offsetof
PID thread_offsetof (pid)
TID thread_offsetof (tid)
POINTER_GUARD (offsetof (tcbhead_t, pointer_guard) - TLS_TCB_OFFSET - sizeof (tcbhead_t))
+#ifndef __ASSUME_PRIVATE_FUTEX
+PRIVATE_FUTEX_OFFSET thread_offsetof (header.private_futex)
+#endif
--- libc/nptl/sysdeps/pthread/aio_misc.h 5 Sep 2006 15:22:18 -0000 1.3
+++ libc/nptl/sysdeps/pthread/aio_misc.h 28 Jul 2007 19:30:21 -0000 1.5
@@ -30,7 +30,7 @@
#define AIO_MISC_NOTIFY(waitlist) \
do { \
if (*waitlist->counterp > 0 && --*waitlist->counterp == 0) \
- lll_futex_wake (waitlist->counterp, 1); \
+ lll_futex_wake (waitlist->counterp, 1, LLL_PRIVATE); \
} while (0)
#define AIO_MISC_WAIT(result, futex, timeout, cancel) \
@@ -49,7 +49,8 @@
int status; \
do \
{ \
- status = lll_futex_timed_wait (futexaddr, oldval, timeout); \
+ status = lll_futex_timed_wait (futexaddr, oldval, timeout, \
+ LLL_PRIVATE); \
if (status != -EWOULDBLOCK) \
break; \
\
--- libc/nptl/sysdeps/pthread/createthread.c 5 Sep 2006 17:13:14 -0000 1.28
+++ libc/nptl/sysdeps/pthread/createthread.c 1 Aug 2007 04:20:14 -0000 1.29
@@ -60,7 +60,7 @@ do_clone (struct pthread *pd, const stru
/* We Make sure the thread does not run far by forcing it to get a
lock. We lock it here too so that the new thread cannot continue
until we tell it to. */
- lll_lock (pd->lock);
+ lll_lock (pd->lock, LLL_PRIVATE);
/* One more thread. We cannot have the thread do this itself, since it
might exist but not have been scheduled yet by the time we've returned
@@ -223,7 +223,7 @@ create_thread (struct pthread *pd, const
__nptl_create_event ();
/* And finally restart the new thread. */
- lll_unlock (pd->lock);
+ lll_unlock (pd->lock, LLL_PRIVATE);
}
return res;
@@ -250,7 +250,7 @@ create_thread (struct pthread *pd, const
if (res == 0 && stopped)
/* And finally restart the new thread. */
- lll_unlock (pd->lock);
+ lll_unlock (pd->lock, LLL_PRIVATE);
return res;
}
--- libc/nptl/sysdeps/pthread/gai_misc.h 5 Sep 2006 15:23:12 -0000 1.2
+++ libc/nptl/sysdeps/pthread/gai_misc.h 28 Jul 2007 19:30:21 -0000 1.4
@@ -31,7 +31,7 @@
#define GAI_MISC_NOTIFY(waitlist) \
do { \
if (*waitlist->counterp > 0 && --*waitlist->counterp == 0) \
- lll_futex_wake (waitlist->counterp, 1); \
+ lll_futex_wake (waitlist->counterp, 1, LLL_PRIVATE); \
} while (0)
#define GAI_MISC_WAIT(result, futex, timeout, cancel) \
@@ -50,7 +50,8 @@
int status; \
do \
{ \
- status = lll_futex_timed_wait (futexaddr, oldval, timeout); \
+ status = lll_futex_timed_wait (futexaddr, oldval, timeout, \
+ LLL_PRIVATE); \
if (status != -EWOULDBLOCK) \
break; \
\
--- libc/nptl/sysdeps/pthread/pthread.h 17 Mar 2007 17:04:07 -0000 1.43
+++ libc/nptl/sysdeps/pthread/pthread.h 27 May 2007 18:03:27 -0000 1.44
@@ -20,6 +20,7 @@
#define _PTHREAD_H 1
#include <features.h>
+#include <endian.h>
#include <sched.h>
#include <time.h>
@@ -119,21 +120,23 @@ enum
};
/* Read-write lock initializers. */
-# if __WORDSIZE == 64
-# define PTHREAD_RWLOCK_INITIALIZER \
+# define PTHREAD_RWLOCK_INITIALIZER \
{ { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 } }
-# else
-# define PTHREAD_RWLOCK_INITIALIZER \
- { { 0, 0, 0, 0, 0, 0, 0, 0 } }
-# endif
# ifdef __USE_GNU
# if __WORDSIZE == 64
# define PTHREAD_RWLOCK_WRITER_NONRECURSIVE_INITIALIZER_NP \
{ { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, \
PTHREAD_RWLOCK_PREFER_WRITER_NONRECURSIVE_NP } }
# else
-# define PTHREAD_RWLOCK_WRITER_NONRECURSIVE_INITIALIZER_NP \
- { { 0, 0, 0, 0, 0, 0, PTHREAD_RWLOCK_PREFER_WRITER_NONRECURSIVE_NP, 0 } }
+# if __BYTE_ORDER == __LITTLE_ENDIAN
+# define PTHREAD_RWLOCK_WRITER_NONRECURSIVE_INITIALIZER_NP \
+ { { 0, 0, 0, 0, 0, 0, PTHREAD_RWLOCK_PREFER_WRITER_NONRECURSIVE_NP, \
+ 0, 0, 0, 0 } }
+# else
+# define PTHREAD_RWLOCK_WRITER_NONRECURSIVE_INITIALIZER_NP \
+ { { 0, 0, 0, 0, 0, 0, 0, 0, 0, PTHREAD_RWLOCK_PREFER_WRITER_NONRECURSIVE_NP,\
+ 0 } }
+# endif
# endif
# endif
#endif /* Unix98 or XOpen2K */
--- libc/nptl/sysdeps/pthread/bits/libc-lock.h 17 Mar 2007 17:04:07 -0000 1.21
+++ libc/nptl/sysdeps/pthread/bits/libc-lock.h 10 Oct 2007 15:59:42 -0000 1.23
@@ -235,7 +235,7 @@ typedef pthread_key_t __libc_key_t;
/* Lock the named lock variable. */
#if defined _LIBC && (!defined NOT_IN_libc || defined IS_IN_libpthread)
# define __libc_lock_lock(NAME) \
- ({ lll_lock (NAME); 0; })
+ ({ lll_lock (NAME, LLL_PRIVATE); 0; })
#else
# define __libc_lock_lock(NAME) \
__libc_maybe_call (__pthread_mutex_lock, (&(NAME)), 0)
@@ -252,7 +252,7 @@ typedef pthread_key_t __libc_key_t;
void *self = THREAD_SELF; \
if ((NAME).owner != self) \
{ \
- lll_lock ((NAME).lock); \
+ lll_lock ((NAME).lock, LLL_PRIVATE); \
(NAME).owner = self; \
} \
++(NAME).cnt; \
@@ -306,7 +306,7 @@ typedef pthread_key_t __libc_key_t;
/* Unlock the named lock variable. */
#if defined _LIBC && (!defined NOT_IN_libc || defined IS_IN_libpthread)
# define __libc_lock_unlock(NAME) \
- lll_unlock (NAME)
+ lll_unlock (NAME, LLL_PRIVATE)
#else
# define __libc_lock_unlock(NAME) \
__libc_maybe_call (__pthread_mutex_unlock, (&(NAME)), 0)
@@ -322,7 +322,7 @@ typedef pthread_key_t __libc_key_t;
if (--(NAME).cnt == 0) \
{ \
(NAME).owner = NULL; \
- lll_unlock ((NAME).lock); \
+ lll_unlock ((NAME).lock, LLL_PRIVATE); \
} \
} while (0)
#else
--- libc/nptl/sysdeps/pthread/bits/stdio-lock.h 4 Nov 2003 23:43:10 -0000 1.5
+++ libc/nptl/sysdeps/pthread/bits/stdio-lock.h 1 Aug 2007 04:20:35 -0000 1.7
@@ -42,7 +42,7 @@ typedef struct { int lock; int cnt; void
void *__self = THREAD_SELF; \
if ((_name).owner != __self) \
{ \
- lll_lock ((_name).lock); \
+ lll_lock ((_name).lock, LLL_PRIVATE); \
(_name).owner = __self; \
} \
++(_name).cnt; \
@@ -72,7 +72,7 @@ typedef struct { int lock; int cnt; void
if (--(_name).cnt == 0) \
{ \
(_name).owner = NULL; \
- lll_unlock ((_name).lock); \
+ lll_unlock ((_name).lock, LLL_PRIVATE); \
} \
} while (0)
--- libc/nptl/sysdeps/s390/tls.h 27 Oct 2006 23:11:44 -0000 1.15
+++ libc/nptl/sysdeps/s390/tls.h 28 Jul 2007 19:30:20 -0000 1.18
@@ -27,6 +27,7 @@
# include <stdint.h>
# include <stdlib.h>
# include <list.h>
+# include <kernel-features.h>
/* Type for the dtv. */
@@ -51,6 +52,9 @@ typedef struct
uintptr_t sysinfo;
uintptr_t stack_guard;
int gscope_flag;
+#ifndef __ASSUME_PRIVATE_FUTEX
+ int private_futex;
+#endif
} tcbhead_t;
# ifndef __s390x__
@@ -182,7 +186,7 @@ typedef struct
= atomic_exchange_rel (&THREAD_SELF->header.gscope_flag, \
THREAD_GSCOPE_FLAG_UNUSED); \
if (__res == THREAD_GSCOPE_FLAG_WAIT) \
- lll_futex_wake (&THREAD_SELF->header.gscope_flag, 1); \
+ lll_futex_wake (&THREAD_SELF->header.gscope_flag, 1, LLL_PRIVATE); \
} \
while (0)
#define THREAD_GSCOPE_SET_FLAG() \
--- libc/nptl/sysdeps/unix/sysv/linux/Makefile 29 Jul 2006 04:33:52 -0000 1.15
+++ libc/nptl/sysdeps/unix/sysv/linux/Makefile 26 May 2007 01:30:30 -0000 1.16
@@ -25,7 +25,8 @@ libpthread-sysdep_routines += pt-fork pt
gen-as-const-headers += lowlevelcond.sym lowlevelrwlock.sym \
lowlevelbarrier.sym unwindbuf.sym \
- lowlevelrobustlock.sym pthread-pi-defines.sym
+ lowlevelrobustlock.sym pthread-pi-defines.sym \
+ structsem.sym
endif
ifeq ($(subdir),posix)
--- libc/nptl/sysdeps/unix/sysv/linux/fork.c 25 Feb 2007 21:43:28 -0000 1.12
+++ libc/nptl/sysdeps/unix/sysv/linux/fork.c 1 Aug 2007 04:22:51 -0000 1.15
@@ -183,7 +183,7 @@ __libc_fork (void)
}
/* Initialize the fork lock. */
- __fork_lock = (lll_lock_t) LLL_LOCK_INITIALIZER;
+ __fork_lock = LLL_LOCK_INITIALIZER;
}
else
{
@@ -203,7 +203,7 @@ __libc_fork (void)
if (atomic_decrement_and_test (&allp->handler->refcntr)
&& allp->handler->need_signal)
- lll_futex_wake (allp->handler->refcntr, 1);
+ lll_futex_wake (allp->handler->refcntr, 1, LLL_PRIVATE);
allp = allp->next;
}
--- libc/nptl/sysdeps/unix/sysv/linux/fork.h 15 May 2006 20:19:43 -0000 1.9
+++ libc/nptl/sysdeps/unix/sysv/linux/fork.h 28 Oct 2007 00:59:56 -0000 1.11
@@ -26,7 +26,7 @@ extern unsigned long int __fork_generati
extern unsigned long int *__fork_generation_pointer attribute_hidden;
/* Lock to protect allocation and deallocation of fork handlers. */
-extern lll_lock_t __fork_lock attribute_hidden;
+extern int __fork_lock attribute_hidden;
/* Elements of the fork handler lists. */
struct fork_handler
--- libc/nptl/sysdeps/unix/sysv/linux/internaltypes.h 2 Sep 2004 18:50:02 -0000 1.8
+++ libc/nptl/sysdeps/unix/sysv/linux/internaltypes.h 29 May 2007 21:27:43 -0000 1.12
@@ -76,9 +76,11 @@ struct pthread_condattr
/* The __NWAITERS field is used as a counter and to house the number
- of bits which represent the clock. COND_CLOCK_BITS is the number
- of bits reserved for the clock. */
-#define COND_CLOCK_BITS 1
+ of bits for other purposes. COND_CLOCK_BITS is the number
+ of bits needed to represent the ID of the clock. COND_NWAITERS_SHIFT
+ is the number of bits reserved for other purposes like the clock. */
+#define COND_CLOCK_BITS 1
+#define COND_NWAITERS_SHIFT 1
/* Read-write lock variable attribute data structure. */
@@ -96,6 +98,7 @@ struct pthread_barrier
int lock;
unsigned int left;
unsigned int init_count;
+ int private;
};
@@ -137,9 +140,16 @@ struct pthread_key_struct
/* Semaphore variable structure. */
-struct sem
+struct new_sem
{
- unsigned int count;
+ unsigned int value;
+ int private;
+ unsigned long int nwaiters;
+};
+
+struct old_sem
+{
+ unsigned int value;
};
--- libc/nptl/sysdeps/unix/sysv/linux/lowlevelbarrier.sym 11 Mar 2003 05:24:34 -0000 1.1
+++ libc/nptl/sysdeps/unix/sysv/linux/lowlevelbarrier.sym 26 May 2007 16:18:57 -0000 1.2
@@ -9,3 +9,4 @@ CURR_EVENT offsetof (struct pthread_bar
MUTEX offsetof (struct pthread_barrier, lock)
LEFT offsetof (struct pthread_barrier, left)
INIT_COUNT offsetof (struct pthread_barrier, init_count)
+PRIVATE offsetof (struct pthread_barrier, private)
--- libc/nptl/sysdeps/unix/sysv/linux/lowlevelcond.sym 2 Sep 2004 18:50:27 -0000 1.6
+++ libc/nptl/sysdeps/unix/sysv/linux/lowlevelcond.sym 29 May 2007 16:15:39 -0000 1.7
@@ -13,4 +13,4 @@ wakeup_seq offsetof (pthread_cond_t, __d
woken_seq offsetof (pthread_cond_t, __data.__woken_seq)
dep_mutex offsetof (pthread_cond_t, __data.__mutex)
broadcast_seq offsetof (pthread_cond_t, __data.__broadcast_seq)
-clock_bits COND_CLOCK_BITS
+nwaiters_shift COND_NWAITERS_SHIFT
--- libc/nptl/sysdeps/unix/sysv/linux/lowlevellock.c 22 Sep 2003 21:25:15 -0000 1.13
+++ libc/nptl/sysdeps/unix/sysv/linux/lowlevellock.c 14 Aug 2007 19:59:34 -0000 1.17
@@ -25,20 +25,35 @@
void
-__lll_lock_wait (int *futex)
+__lll_lock_wait_private (int *futex)
{
do
{
int oldval = atomic_compare_and_exchange_val_acq (futex, 2, 1);
if (oldval != 0)
- lll_futex_wait (futex, 2);
+ lll_futex_wait (futex, 2, LLL_PRIVATE);
+ }
+ while (atomic_compare_and_exchange_bool_acq (futex, 2, 0) != 0);
+}
+
+
+/* These functions don't get included in libc.so */
+#ifdef IS_IN_libpthread
+void
+__lll_lock_wait (int *futex, int private)
+{
+ do
+ {
+ int oldval = atomic_compare_and_exchange_val_acq (futex, 2, 1);
+ if (oldval != 0)
+ lll_futex_wait (futex, 2, private);
}
while (atomic_compare_and_exchange_bool_acq (futex, 2, 0) != 0);
}
int
-__lll_timedlock_wait (int *futex, const struct timespec *abstime)
+__lll_timedlock_wait (int *futex, const struct timespec *abstime, int private)
{
/* Reject invalid timeouts. */
if (abstime->tv_nsec < 0 || abstime->tv_nsec >= 1000000000)
@@ -68,7 +83,7 @@ __lll_timedlock_wait (int *futex, const
/* Wait. */
int oldval = atomic_compare_and_exchange_val_acq (futex, 2, 1);
if (oldval != 0)
- lll_futex_timed_wait (futex, 2, &rt);
+ lll_futex_timed_wait (futex, 2, &rt, private);
}
while (atomic_compare_and_exchange_bool_acq (futex, 2, 0) != 0);
@@ -76,20 +91,6 @@ __lll_timedlock_wait (int *futex, const
}
-/* These don't get included in libc.so */
-#ifdef IS_IN_libpthread
-int
-lll_unlock_wake_cb (int *futex)
-{
- int val = atomic_exchange_rel (futex, 0);
-
- if (__builtin_expect (val > 1, 0))
- lll_futex_wake (futex, 1);
-
- return 0;
-}
-
-
int
__lll_timedwait_tid (int *tidp, const struct timespec *abstime)
{
@@ -120,12 +121,12 @@ __lll_timedwait_tid (int *tidp, const st
if (rt.tv_sec < 0)
return ETIMEDOUT;
- /* Wait until thread terminates. */
- if (lll_futex_timed_wait (tidp, tid, &rt) == -ETIMEDOUT)
+ /* Wait until thread terminates. The kernel so far does not use
+ the private futex operations for this. */
+ if (lll_futex_timed_wait (tidp, tid, &rt, LLL_SHARED) == -ETIMEDOUT)
return ETIMEDOUT;
}
return 0;
}
-
#endif
--- libc/nptl/sysdeps/unix/sysv/linux/lowlevelrobustlock.c 7 May 2007 14:25:01 -0000 1.4
+++ libc/nptl/sysdeps/unix/sysv/linux/lowlevelrobustlock.c 1 Aug 2007 04:32:17 -0000 1.6
@@ -25,7 +25,7 @@
int
-__lll_robust_lock_wait (int *futex)
+__lll_robust_lock_wait (int *futex, int private)
{
int oldval = *futex;
int tid = THREAD_GETMEM (THREAD_SELF, tid);
@@ -44,7 +44,7 @@ __lll_robust_lock_wait (int *futex)
&& atomic_compare_and_exchange_bool_acq (futex, newval, oldval))
continue;
- lll_futex_wait (futex, newval);
+ lll_futex_wait (futex, newval, private);
try:
;
@@ -57,7 +57,8 @@ __lll_robust_lock_wait (int *futex)
int
-__lll_robust_timedlock_wait (int *futex, const struct timespec *abstime)
+__lll_robust_timedlock_wait (int *futex, const struct timespec *abstime,
+ int private)
{
/* Reject invalid timeouts. */
if (abstime->tv_nsec < 0 || abstime->tv_nsec >= 1000000000)
@@ -100,7 +101,7 @@ __lll_robust_timedlock_wait (int *futex,
&& atomic_compare_and_exchange_bool_acq (futex, newval, oldval))
continue;
- lll_futex_timed_wait (futex, newval, &rt);
+ lll_futex_timed_wait (futex, newval, &rt, private);
try:
;
--- libc/nptl/sysdeps/unix/sysv/linux/lowlevelrwlock.sym 10 Mar 2003 21:00:28 -0000 1.1
+++ libc/nptl/sysdeps/unix/sysv/linux/lowlevelrwlock.sym 27 May 2007 18:04:02 -0000 1.2
@@ -1,6 +1,7 @@
#include <stddef.h>
#include <stdio.h>
#include <bits/pthreadtypes.h>
+#include <bits/wordsize.h>
--
@@ -12,3 +13,4 @@ READERS_QUEUED offsetof (pthread_rwlock_
WRITERS_QUEUED offsetof (pthread_rwlock_t, __data.__nr_writers_queued)
FLAGS offsetof (pthread_rwlock_t, __data.__flags)
WRITER offsetof (pthread_rwlock_t, __data.__writer)
+PSHARED offsetof (pthread_rwlock_t, __data.__shared)
--- libc/nptl/sysdeps/unix/sysv/linux/pthread-pi-defines.sym 29 Jul 2006 04:33:20 -0000 1.1
+++ libc/nptl/sysdeps/unix/sysv/linux/pthread-pi-defines.sym 11 Aug 2007 18:48:00 -0000 1.2
@@ -4,3 +4,4 @@
MUTEX_KIND offsetof (pthread_mutex_t, __data.__kind)
PI_BIT PTHREAD_MUTEX_PRIO_INHERIT_NP
+PS_BIT PTHREAD_MUTEX_PSHARED_BIT
--- libc/nptl/sysdeps/unix/sysv/linux/pthread_mutex_cond_lock.c 15 Feb 2006 17:01:17 -0000 1.4
+++ libc/nptl/sysdeps/unix/sysv/linux/pthread_mutex_cond_lock.c 11 Aug 2007 18:50:38 -0000 1.6
@@ -1,8 +1,12 @@
#include <pthreadP.h>
-#define LLL_MUTEX_LOCK(mutex) lll_mutex_cond_lock (mutex)
-#define LLL_MUTEX_TRYLOCK(mutex) lll_mutex_cond_trylock (mutex)
-#define LLL_ROBUST_MUTEX_LOCK(mutex, id) lll_robust_mutex_cond_lock (mutex, id)
+#define LLL_MUTEX_LOCK(mutex) \
+ lll_cond_lock ((mutex)->__data.__lock, PTHREAD_MUTEX_PSHARED (mutex))
+#define LLL_MUTEX_TRYLOCK(mutex) \
+ lll_cond_trylock ((mutex)->__data.__lock)
+#define LLL_ROBUST_MUTEX_LOCK(mutex, id) \
+ lll_robust_cond_lock ((mutex)->__data.__lock, id, \
+ PTHREAD_ROBUST_MUTEX_PSHARED (mutex))
#define __pthread_mutex_lock __pthread_mutex_cond_lock
#define NO_INCR
--- libc/nptl/sysdeps/unix/sysv/linux/register-atfork.c 21 Dec 2005 22:17:21 -0000 1.9
+++ libc/nptl/sysdeps/unix/sysv/linux/register-atfork.c 28 Oct 2007 01:00:08 -0000 1.11
@@ -24,7 +24,7 @@
/* Lock to protect allocation and deallocation of fork handlers. */
-lll_lock_t __fork_lock = LLL_LOCK_INITIALIZER;
+int __fork_lock = LLL_LOCK_INITIALIZER;
/* Number of pre-allocated handler entries. */
@@ -85,7 +85,7 @@ __register_atfork (prepare, parent, chil
void *dso_handle;
{
/* Get the lock to not conflict with other allocations. */
- lll_lock (__fork_lock);
+ lll_lock (__fork_lock, LLL_PRIVATE);
struct fork_handler *newp = fork_handler_alloc ();
@@ -102,7 +102,7 @@ __register_atfork (prepare, parent, chil
}
/* Release the lock. */
- lll_unlock (__fork_lock);
+ lll_unlock (__fork_lock, LLL_PRIVATE);
return newp == NULL ? ENOMEM : 0;
}
@@ -112,7 +112,7 @@ libc_hidden_def (__register_atfork)
libc_freeres_fn (free_mem)
{
/* Get the lock to not conflict with running forks. */
- lll_lock (__fork_lock);
+ lll_lock (__fork_lock, LLL_PRIVATE);
/* No more fork handlers. */
__fork_handlers = NULL;
@@ -123,7 +123,7 @@ libc_freeres_fn (free_mem)
memset (&fork_handler_pool, '\0', sizeof (fork_handler_pool));
/* Release the lock. */
- lll_unlock (__fork_lock);
+ lll_unlock (__fork_lock, LLL_PRIVATE);
/* We can free the memory after releasing the lock. */
while (runp != NULL)
--- libc/nptl/sysdeps/unix/sysv/linux/sem_post.c 15 May 2007 06:31:57 -0000 1.6
+++ libc/nptl/sysdeps/unix/sysv/linux/sem_post.c 1 Aug 2007 04:32:02 -0000 1.9
@@ -29,10 +29,35 @@
int
__new_sem_post (sem_t *sem)
{
+ struct new_sem *isem = (struct new_sem *) sem;
+
+ int nr = atomic_increment_val (&isem->value);
+ atomic_full_barrier ();
+ if (isem->nwaiters > 0)
+ {
+ int err = lll_futex_wake (&isem->value, 1,
+ isem->private ^ FUTEX_PRIVATE_FLAG);
+ if (__builtin_expect (err, 0) < 0)
+ {
+ __set_errno (-err);
+ return -1;
+ }
+ }
+ return 0;
+}
+versioned_symbol (libpthread, __new_sem_post, sem_post, GLIBC_2_1);
+
+
+#if SHLIB_COMPAT (libpthread, GLIBC_2_0, GLIBC_2_1)
+int
+attribute_compat_text_section
+__old_sem_post (sem_t *sem)
+{
int *futex = (int *) sem;
int nr = atomic_increment_val (futex);
- int err = lll_futex_wake (futex, nr);
+ /* We always have to assume it is a shared semaphore. */
+ int err = lll_futex_wake (futex, 1, LLL_SHARED);
if (__builtin_expect (err, 0) < 0)
{
__set_errno (-err);
@@ -40,8 +65,5 @@ __new_sem_post (sem_t *sem)
}
return 0;
}
-versioned_symbol (libpthread, __new_sem_post, sem_post, GLIBC_2_1);
-#if SHLIB_COMPAT (libpthread, GLIBC_2_0, GLIBC_2_1)
-strong_alias (__new_sem_post, __old_sem_post)
compat_symbol (libpthread, __old_sem_post, sem_post, GLIBC_2_0);
#endif
--- libc/nptl/sysdeps/unix/sysv/linux/sem_timedwait.c 8 Jun 2003 05:20:02 -0000 1.4
+++ libc/nptl/sysdeps/unix/sysv/linux/sem_timedwait.c 1 Aug 2007 04:42:06 -0000 1.8
@@ -28,28 +28,29 @@
#include <shlib-compat.h>
+extern void __sem_wait_cleanup (void *arg) attribute_hidden;
+
+
int
sem_timedwait (sem_t *sem, const struct timespec *abstime)
{
- /* First check for cancellation. */
- CANCELLATION_P (THREAD_SELF);
-
- int *futex = (int *) sem;
- int val;
+ struct new_sem *isem = (struct new_sem *) sem;
int err;
- if (*futex > 0)
+ if (atomic_decrement_if_positive (&isem->value) > 0)
+ return 0;
+
+ if (abstime->tv_nsec < 0 || abstime->tv_nsec >= 1000000000)
{
- val = atomic_decrement_if_positive (futex);
- if (val > 0)
- return 0;
+ __set_errno (EINVAL);
+ return -1;
}
- err = -EINVAL;
- if (abstime->tv_nsec < 0 || abstime->tv_nsec >= 1000000000)
- goto error_return;
+ atomic_increment (&isem->nwaiters);
+
+ pthread_cleanup_push (__sem_wait_cleanup, isem);
- do
+ while (1)
{
struct timeval tv;
struct timespec rt;
@@ -70,7 +71,11 @@ sem_timedwait (sem_t *sem, const struct
/* Already timed out? */
err = -ETIMEDOUT;
if (sec < 0)
- goto error_return;
+ {
+ __set_errno (ETIMEDOUT);
+ err = -1;
+ break;
+ }
/* Do wait. */
rt.tv_sec = sec;
@@ -79,21 +84,29 @@ sem_timedwait (sem_t *sem, const struct
/* Enable asynchronous cancellation. Required by the standard. */
int oldtype = __pthread_enable_asynccancel ();
- err = lll_futex_timed_wait (futex, 0, &rt);
+ err = lll_futex_timed_wait (&isem->value, 0, &rt,
+ isem->private ^ FUTEX_PRIVATE_FLAG);
/* Disable asynchronous cancellation. */
__pthread_disable_asynccancel (oldtype);
if (err != 0 && err != -EWOULDBLOCK)
- goto error_return;
+ {
+ __set_errno (-err);
+ err = -1;
+ break;
+ }
- val = atomic_decrement_if_positive (futex);
+ if (atomic_decrement_if_positive (&isem->value) > 0)
+ {
+ err = 0;
+ break;
+ }
}
- while (val <= 0);
- return 0;
+ pthread_cleanup_pop (0);
+
+ atomic_decrement (&isem->nwaiters);
- error_return:
- __set_errno (-err);
- return -1;
+ return err;
}
--- libc/nptl/sysdeps/unix/sysv/linux/sem_wait.c 17 May 2007 18:39:12 -0000 1.5
+++ libc/nptl/sysdeps/unix/sysv/linux/sem_wait.c 1 Aug 2007 04:46:43 -0000 1.9
@@ -28,12 +28,68 @@
#include <shlib-compat.h>
+void
+attribute_hidden
+__sem_wait_cleanup (void *arg)
+{
+ struct new_sem *isem = (struct new_sem *) arg;
+
+ atomic_decrement (&isem->nwaiters);
+}
+
+
int
__new_sem_wait (sem_t *sem)
{
- /* First check for cancellation. */
- CANCELLATION_P (THREAD_SELF);
+ struct new_sem *isem = (struct new_sem *) sem;
+ int err;
+
+ if (atomic_decrement_if_positive (&isem->value) > 0)
+ return 0;
+
+ atomic_increment (&isem->nwaiters);
+
+ pthread_cleanup_push (__sem_wait_cleanup, isem);
+
+ while (1)
+ {
+ /* Enable asynchronous cancellation. Required by the standard. */
+ int oldtype = __pthread_enable_asynccancel ();
+
+ err = lll_futex_wait (&isem->value, 0,
+ isem->private ^ FUTEX_PRIVATE_FLAG);
+
+ /* Disable asynchronous cancellation. */
+ __pthread_disable_asynccancel (oldtype);
+
+ if (err != 0 && err != -EWOULDBLOCK)
+ {
+ __set_errno (-err);
+ err = -1;
+ break;
+ }
+
+ if (atomic_decrement_if_positive (&isem->value) > 0)
+ {
+ err = 0;
+ break;
+ }
+ }
+ pthread_cleanup_pop (0);
+
+ atomic_decrement (&isem->nwaiters);
+
+ return err;
+}
+versioned_symbol (libpthread, __new_sem_wait, sem_wait, GLIBC_2_1);
+
+
+#if SHLIB_COMPAT (libpthread, GLIBC_2_0, GLIBC_2_1)
+int
+attribute_compat_text_section
+__old_sem_wait (sem_t *sem)
+{
int *futex = (int *) sem;
int err;
@@ -45,7 +101,8 @@ __new_sem_wait (sem_t *sem)
/* Enable asynchronous cancellation. Required by the standard. */
int oldtype = __pthread_enable_asynccancel ();
- err = lll_futex_wait (futex, 0);
+ /* Always assume the semaphore is shared. */
+ err = lll_futex_wait (futex, 0, LLL_SHARED);
/* Disable asynchronous cancellation. */
__pthread_disable_asynccancel (oldtype);
@@ -56,8 +113,5 @@ __new_sem_wait (sem_t *sem)
return -1;
}
-versioned_symbol (libpthread, __new_sem_wait, sem_wait, GLIBC_2_1);
-#if SHLIB_COMPAT (libpthread, GLIBC_2_0, GLIBC_2_1)
-strong_alias (__new_sem_wait, __old_sem_wait)
compat_symbol (libpthread, __old_sem_wait, sem_wait, GLIBC_2_0);
#endif
--- libc/nptl/sysdeps/unix/sysv/linux/structsem.sym 1 Jan 1970 00:00:00 -0000
+++ libc/nptl/sysdeps/unix/sysv/linux/structsem.sym 26 May 2007 01:24:50 -0000 1.1
@@ -0,0 +1,10 @@
+#include <stddef.h>
+#include <sched.h>
+#include <bits/pthreadtypes.h>
+#include "internaltypes.h"
+
+--
+
+VALUE offsetof (struct new_sem, value)
+PRIVATE offsetof (struct new_sem, private)
+NWAITERS offsetof (struct new_sem, nwaiters)
--- libc/nptl/sysdeps/unix/sysv/linux/unregister-atfork.c 21 May 2007 18:24:37 -0000 1.8
+++ libc/nptl/sysdeps/unix/sysv/linux/unregister-atfork.c 28 Oct 2007 01:00:21 -0000 1.12
@@ -54,7 +54,7 @@ __unregister_atfork (dso_handle)
that there couldn't have been another thread deleting something.
The __unregister_atfork function is only called from the
dlclose() code which itself serializes the operations. */
- lll_lock (__fork_lock);
+ lll_lock (__fork_lock, LLL_PRIVATE);
/* We have to create a new list with all the entries we don't remove. */
struct deleted_handler
@@ -89,7 +89,7 @@ __unregister_atfork (dso_handle)
while (runp != NULL);
/* Release the lock. */
- lll_unlock (__fork_lock);
+ lll_unlock (__fork_lock, LLL_PRIVATE);
/* Walk the list of all entries which have to be deleted. */
while (deleted != NULL)
@@ -104,7 +104,7 @@ __unregister_atfork (dso_handle)
atomic_decrement (&deleted->handler->refcntr);
unsigned int val;
while ((val = deleted->handler->refcntr) != 0)
- lll_futex_wait (&deleted->handler->refcntr, val);
+ lll_futex_wait (&deleted->handler->refcntr, val, LLL_PRIVATE);
deleted = deleted->next;
}
--- libc/nptl/sysdeps/unix/sysv/linux/i386/lowlevellock.h 5 Sep 2006 14:44:25 -0000 1.32
+++ libc/nptl/sysdeps/unix/sysv/linux/i386/lowlevellock.h 14 Aug 2007 02:24:10 -0000 1.38
@@ -20,30 +20,81 @@
#ifndef _LOWLEVELLOCK_H
#define _LOWLEVELLOCK_H 1
-#include <time.h>
-#include <sys/param.h>
-#include <bits/pthreadtypes.h>
-
-#ifndef LOCK_INSTR
-# ifdef UP
-# define LOCK_INSTR /* nothing */
-# else
-# define LOCK_INSTR "lock;"
+#ifndef __ASSEMBLER__
+# include <time.h>
+# include <sys/param.h>
+# include <bits/pthreadtypes.h>
+# include <kernel-features.h>
+# include <tcb-offsets.h>
+
+# ifndef LOCK_INSTR
+# ifdef UP
+# define LOCK_INSTR /* nothing */
+# else
+# define LOCK_INSTR "lock;"
+# endif
+# endif
+#else
+# ifndef LOCK
+# ifdef UP
+# define LOCK
+# else
+# define LOCK lock
+# endif
# endif
#endif
#define SYS_futex 240
#define FUTEX_WAIT 0
#define FUTEX_WAKE 1
+#define FUTEX_CMP_REQUEUE 4
+#define FUTEX_WAKE_OP 5
#define FUTEX_LOCK_PI 6
#define FUTEX_UNLOCK_PI 7
#define FUTEX_TRYLOCK_PI 8
+#define FUTEX_PRIVATE_FLAG 128
+#define FUTEX_OP_CLEAR_WAKE_IF_GT_ONE ((4 << 24) | 1)
+
+/* Values for 'private' parameter of locking macros. Yes, the
+ definition seems to be backwards. But it is not. The bit will be
+ reversed before passing to the system call. */
+#define LLL_PRIVATE 0
+#define LLL_SHARED FUTEX_PRIVATE_FLAG
+
+
+#if !defined NOT_IN_libc || defined IS_IN_rtld
+/* In libc.so or ld.so all futexes are private. */
+# ifdef __ASSUME_PRIVATE_FUTEX
+# define __lll_private_flag(fl, private) \
+ ((fl) | FUTEX_PRIVATE_FLAG)
+# else
+# define __lll_private_flag(fl, private) \
+ ((fl) | THREAD_GETMEM (THREAD_SELF, header.private_futex))
+# endif
+#else
+# ifdef __ASSUME_PRIVATE_FUTEX
+# define __lll_private_flag(fl, private) \
+ (((fl) | FUTEX_PRIVATE_FLAG) ^ (private))
+# else
+# define __lll_private_flag(fl, private) \
+ (__builtin_constant_p (private) \
+ ? ((private) == 0 \
+ ? ((fl) | THREAD_GETMEM (THREAD_SELF, header.private_futex)) \
+ : (fl)) \
+ : ({ unsigned int __fl = ((private) ^ FUTEX_PRIVATE_FLAG); \
+ asm ("andl %%gs:%P1, %0" : "+r" (__fl) \
+ : "i" (offsetof (struct pthread, header.private_futex))); \
+ __fl | (fl); }))
+# endif
+#endif
+
+#ifndef __ASSEMBLER__
/* Initializer for compatibility lock. */
-#define LLL_MUTEX_LOCK_INITIALIZER (0)
-#define LLL_MUTEX_LOCK_INITIALIZER_LOCKED (1)
-#define LLL_MUTEX_LOCK_INITIALIZER_WAITERS (2)
+#define LLL_LOCK_INITIALIZER (0)
+#define LLL_LOCK_INITIALIZER_LOCKED (1)
+#define LLL_LOCK_INITIALIZER_WAITERS (2)
#ifdef PIC
@@ -65,7 +116,7 @@
#endif
/* Delay in spinlock loop. */
-#define BUSY_WAIT_NOP asm ("rep; nop")
+#define BUSY_WAIT_NOP asm ("rep; nop")
#define LLL_STUB_UNWIND_INFO_START \
@@ -144,23 +195,11 @@ LLL_STUB_UNWIND_INFO_START \
LLL_STUB_UNWIND_INFO_END
-#define lll_futex_wait(futex, val) \
- ({ \
- int __status; \
- register __typeof (val) _val asm ("edx") = (val); \
- __asm __volatile (LLL_EBX_LOAD \
- LLL_ENTER_KERNEL \
- LLL_EBX_LOAD \
- : "=a" (__status) \
- : "0" (SYS_futex), LLL_EBX_REG (futex), "S" (0), \
- "c" (FUTEX_WAIT), "d" (_val), \
- "i" (offsetof (tcbhead_t, sysinfo)) \
- : "memory"); \
- __status; \
- })
+#define lll_futex_wait(futex, val, private) \
+ lll_futex_timed_wait (futex, val, NULL, private)
-#define lll_futex_timed_wait(futex, val, timeout) \
+#define lll_futex_timed_wait(futex, val, timeout, private) \
({ \
int __status; \
register __typeof (val) _val asm ("edx") = (val); \
@@ -169,14 +208,14 @@ LLL_STUB_UNWIND_INFO_END
LLL_EBX_LOAD \
: "=a" (__status) \
: "0" (SYS_futex), LLL_EBX_REG (futex), "S" (timeout), \
- "c" (FUTEX_WAIT), "d" (_val), \
- "i" (offsetof (tcbhead_t, sysinfo)) \
+ "c" (__lll_private_flag (FUTEX_WAIT, private)), \
+ "d" (_val), "i" (offsetof (tcbhead_t, sysinfo)) \
: "memory"); \
__status; \
})
-#define lll_futex_wake(futex, nr) \
+#define lll_futex_wake(futex, nr, private) \
do { \
int __ignore; \
register __typeof (nr) _nr asm ("edx") = (nr); \
@@ -185,354 +224,316 @@ LLL_STUB_UNWIND_INFO_END
LLL_EBX_LOAD \
: "=a" (__ignore) \
: "0" (SYS_futex), LLL_EBX_REG (futex), \
- "c" (FUTEX_WAKE), "d" (_nr), \
+ "c" (__lll_private_flag (FUTEX_WAKE, private)), \
+ "d" (_nr), \
"i" (0) /* phony, to align next arg's number */, \
"i" (offsetof (tcbhead_t, sysinfo))); \
} while (0)
-/* Does not preserve %eax and %ecx. */
-extern int __lll_mutex_lock_wait (int val, int *__futex)
- __attribute ((regparm (2))) attribute_hidden;
-/* Does not preserve %eax, %ecx, and %edx. */
-extern int __lll_mutex_timedlock_wait (int val, int *__futex,
- const struct timespec *abstime)
- __attribute ((regparm (3))) attribute_hidden;
-/* Preserves all registers but %eax. */
-extern int __lll_mutex_unlock_wake (int *__futex)
- __attribute ((regparm (1))) attribute_hidden;
-
-
-/* NB: in the lll_mutex_trylock macro we simply return the value in %eax
+/* NB: in the lll_trylock macro we simply return the value in %eax
after the cmpxchg instruction. In case the operation succeded this
value is zero. In case the operation failed, the cmpxchg instruction
has loaded the current value of the memory work which is guaranteed
to be nonzero. */
-#define lll_mutex_trylock(futex) \
+#if defined NOT_IN_libc || defined UP
+# define __lll_trylock_asm LOCK_INSTR "cmpxchgl %2, %1"
+#else
+# define __lll_trylock_asm "cmpl $0, %%gs:%P5\n\t" \
+ "je 0f\n\t" \
+ "lock\n" \
+ "0:\tcmpxchgl %2, %1"
+#endif
+
+#define lll_trylock(futex) \
({ int ret; \
- __asm __volatile (LOCK_INSTR "cmpxchgl %2, %1" \
+ __asm __volatile (__lll_trylock_asm \
: "=a" (ret), "=m" (futex) \
- : "r" (LLL_MUTEX_LOCK_INITIALIZER_LOCKED), "m" (futex),\
- "0" (LLL_MUTEX_LOCK_INITIALIZER) \
+ : "r" (LLL_LOCK_INITIALIZER_LOCKED), "m" (futex), \
+ "0" (LLL_LOCK_INITIALIZER), \
+ "i" (MULTIPLE_THREADS_OFFSET) \
: "memory"); \
ret; })
-
-#define lll_robust_mutex_trylock(futex, id) \
+#define lll_robust_trylock(futex, id) \
({ int ret; \
__asm __volatile (LOCK_INSTR "cmpxchgl %2, %1" \
: "=a" (ret), "=m" (futex) \
: "r" (id), "m" (futex), \
- "0" (LLL_MUTEX_LOCK_INITIALIZER) \
+ "0" (LLL_LOCK_INITIALIZER) \
: "memory"); \
ret; })
-#define lll_mutex_cond_trylock(futex) \
+#define lll_cond_trylock(futex) \
({ int ret; \
__asm __volatile (LOCK_INSTR "cmpxchgl %2, %1" \
: "=a" (ret), "=m" (futex) \
- : "r" (LLL_MUTEX_LOCK_INITIALIZER_WAITERS), \
- "m" (futex), "0" (LLL_MUTEX_LOCK_INITIALIZER) \
+ : "r" (LLL_LOCK_INITIALIZER_WAITERS), \
+ "m" (futex), "0" (LLL_LOCK_INITIALIZER) \
: "memory"); \
ret; })
+#if defined NOT_IN_libc || defined UP
+# define __lll_lock_asm_start LOCK_INSTR "cmpxchgl %1, %2\n\t"
+#else
+# define __lll_lock_asm_start "cmpl $0, %%gs:%P6\n\t" \
+ "je 0f\n\t" \
+ "lock\n" \
+ "0:\tcmpxchgl %1, %2\n\t"
+#endif
-#define lll_mutex_lock(futex) \
- (void) ({ int ignore1, ignore2; \
- __asm __volatile (LOCK_INSTR "cmpxchgl %1, %2\n\t" \
- "jnz _L_mutex_lock_%=\n\t" \
- ".subsection 1\n\t" \
- ".type _L_mutex_lock_%=,@function\n" \
- "_L_mutex_lock_%=:\n" \
- "1:\tleal %2, %%ecx\n" \
- "2:\tcall __lll_mutex_lock_wait\n" \
- "3:\tjmp 18f\n" \
- "4:\t.size _L_mutex_lock_%=, 4b-1b\n\t" \
- ".previous\n" \
- LLL_STUB_UNWIND_INFO_3 \
- "18:" \
- : "=a" (ignore1), "=c" (ignore2), "=m" (futex) \
- : "0" (0), "1" (1), "m" (futex) \
- : "memory"); })
-
+#define lll_lock(futex, private) \
+ (void) \
+ ({ int ignore1, ignore2; \
+ if (__builtin_constant_p (private) && (private) == LLL_PRIVATE) \
+ __asm __volatile (__lll_lock_asm_start \
+ "jnz _L_lock_%=\n\t" \
+ ".subsection 1\n\t" \
+ ".type _L_lock_%=,@function\n" \
+ "_L_lock_%=:\n" \
+ "1:\tleal %2, %%ecx\n" \
+ "2:\tcall __lll_lock_wait_private\n" \
+ "3:\tjmp 18f\n" \
+ "4:\t.size _L_lock_%=, 4b-1b\n\t" \
+ ".previous\n" \
+ LLL_STUB_UNWIND_INFO_3 \
+ "18:" \
+ : "=a" (ignore1), "=c" (ignore2), "=m" (futex) \
+ : "0" (0), "1" (1), "m" (futex), \
+ "i" (MULTIPLE_THREADS_OFFSET) \
+ : "memory"); \
+ else \
+ { \
+ int ignore3; \
+ __asm __volatile (__lll_lock_asm_start \
+ "jnz _L_lock_%=\n\t" \
+ ".subsection 1\n\t" \
+ ".type _L_lock_%=,@function\n" \
+ "_L_lock_%=:\n" \
+ "1:\tleal %2, %%edx\n" \
+ "0:\tmovl %8, %%ecx\n" \
+ "2:\tcall __lll_lock_wait\n" \
+ "3:\tjmp 18f\n" \
+ "4:\t.size _L_lock_%=, 4b-1b\n\t" \
+ ".previous\n" \
+ LLL_STUB_UNWIND_INFO_4 \
+ "18:" \
+ : "=a" (ignore1), "=c" (ignore2), \
+ "=m" (futex), "=&d" (ignore3) \
+ : "1" (1), "m" (futex), \
+ "i" (MULTIPLE_THREADS_OFFSET), "0" (0), \
+ "g" (private) \
+ : "memory"); \
+ } \
+ })
-#define lll_robust_mutex_lock(futex, id) \
- ({ int result, ignore; \
+#define lll_robust_lock(futex, id, private) \
+ ({ int result, ignore1, ignore2; \
__asm __volatile (LOCK_INSTR "cmpxchgl %1, %2\n\t" \
- "jnz _L_robust_mutex_lock_%=\n\t" \
+ "jnz _L_robust_lock_%=\n\t" \
".subsection 1\n\t" \
- ".type _L_robust_mutex_lock_%=,@function\n" \
- "_L_robust_mutex_lock_%=:\n" \
- "1:\tleal %2, %%ecx\n" \
- "2:\tcall __lll_robust_mutex_lock_wait\n" \
+ ".type _L_robust_lock_%=,@function\n" \
+ "_L_robust_lock_%=:\n" \
+ "1:\tleal %2, %%edx\n" \
+ "0:\tmovl %7, %%ecx\n" \
+ "2:\tcall __lll_robust_lock_wait\n" \
"3:\tjmp 18f\n" \
- "4:\t.size _L_robust_mutex_lock_%=, 4b-1b\n\t" \
+ "4:\t.size _L_robust_lock_%=, 4b-1b\n\t" \
".previous\n" \
- LLL_STUB_UNWIND_INFO_3 \
+ LLL_STUB_UNWIND_INFO_4 \
"18:" \
- : "=a" (result), "=c" (ignore), "=m" (futex) \
- : "0" (0), "1" (id), "m" (futex) \
+ : "=a" (result), "=c" (ignore1), "=m" (futex), \
+ "=&d" (ignore2) \
+ : "0" (0), "1" (id), "m" (futex), "g" (private) \
: "memory"); \
result; })
-/* Special version of lll_mutex_lock which causes the unlock function to
+/* Special version of lll_lock which causes the unlock function to
always wakeup waiters. */
-#define lll_mutex_cond_lock(futex) \
- (void) ({ int ignore1, ignore2; \
- __asm __volatile (LOCK_INSTR "cmpxchgl %1, %2\n\t" \
- "jnz _L_mutex_cond_lock_%=\n\t" \
- ".subsection 1\n\t" \
- ".type _L_mutex_cond_lock_%=,@function\n" \
- "_L_mutex_cond_lock_%=:\n" \
- "1:\tleal %2, %%ecx\n" \
- "2:\tcall __lll_mutex_lock_wait\n" \
- "3:\tjmp 18f\n" \
- "4:\t.size _L_mutex_cond_lock_%=, 4b-1b\n\t" \
- ".previous\n" \
- LLL_STUB_UNWIND_INFO_3 \
- "18:" \
- : "=a" (ignore1), "=c" (ignore2), "=m" (futex) \
- : "0" (0), "1" (2), "m" (futex) \
- : "memory"); })
+#define lll_cond_lock(futex, private) \
+ (void) \
+ ({ int ignore1, ignore2, ignore3; \
+ __asm __volatile (LOCK_INSTR "cmpxchgl %1, %2\n\t" \
+ "jnz _L_cond_lock_%=\n\t" \
+ ".subsection 1\n\t" \
+ ".type _L_cond_lock_%=,@function\n" \
+ "_L_cond_lock_%=:\n" \
+ "1:\tleal %2, %%edx\n" \
+ "0:\tmovl %7, %%ecx\n" \
+ "2:\tcall __lll_lock_wait\n" \
+ "3:\tjmp 18f\n" \
+ "4:\t.size _L_cond_lock_%=, 4b-1b\n\t" \
+ ".previous\n" \
+ LLL_STUB_UNWIND_INFO_4 \
+ "18:" \
+ : "=a" (ignore1), "=c" (ignore2), "=m" (futex), \
+ "=&d" (ignore3) \
+ : "0" (0), "1" (2), "m" (futex), "g" (private) \
+ : "memory"); \
+ })
-#define lll_robust_mutex_cond_lock(futex, id) \
- ({ int result, ignore; \
+#define lll_robust_cond_lock(futex, id, private) \
+ ({ int result, ignore1, ignore2; \
__asm __volatile (LOCK_INSTR "cmpxchgl %1, %2\n\t" \
- "jnz _L_robust_mutex_cond_lock_%=\n\t" \
+ "jnz _L_robust_cond_lock_%=\n\t" \
".subsection 1\n\t" \
- ".type _L_robust_mutex_cond_lock_%=,@function\n" \
- "_L_robust_mutex_cond_lock_%=:\n" \
- "1:\tleal %2, %%ecx\n" \
- "2:\tcall __lll_robust_mutex_lock_wait\n" \
+ ".type _L_robust_cond_lock_%=,@function\n" \
+ "_L_robust_cond_lock_%=:\n" \
+ "1:\tleal %2, %%edx\n" \
+ "0:\tmovl %7, %%ecx\n" \
+ "2:\tcall __lll_robust_lock_wait\n" \
"3:\tjmp 18f\n" \
- "4:\t.size _L_robust_mutex_cond_lock_%=, 4b-1b\n\t" \
+ "4:\t.size _L_robust_cond_lock_%=, 4b-1b\n\t" \
".previous\n" \
- LLL_STUB_UNWIND_INFO_3 \
+ LLL_STUB_UNWIND_INFO_4 \
"18:" \
- : "=a" (result), "=c" (ignore), "=m" (futex) \
- : "0" (0), "1" (id | FUTEX_WAITERS), "m" (futex) \
+ : "=a" (result), "=c" (ignore1), "=m" (futex), \
+ "=&d" (ignore2) \
+ : "0" (0), "1" (id | FUTEX_WAITERS), "m" (futex), \
+ "g" (private) \
: "memory"); \
result; })
-#define lll_mutex_timedlock(futex, timeout) \
- ({ int result, ignore1, ignore2; \
+#define lll_timedlock(futex, timeout, private) \
+ ({ int result, ignore1, ignore2, ignore3; \
__asm __volatile (LOCK_INSTR "cmpxchgl %1, %3\n\t" \
- "jnz _L_mutex_timedlock_%=\n\t" \
+ "jnz _L_timedlock_%=\n\t" \
".subsection 1\n\t" \
- ".type _L_mutex_timedlock_%=,@function\n" \
- "_L_mutex_timedlock_%=:\n" \
+ ".type _L_timedlock_%=,@function\n" \
+ "_L_timedlock_%=:\n" \
"1:\tleal %3, %%ecx\n" \
- "0:\tmovl %7, %%edx\n" \
- "2:\tcall __lll_mutex_timedlock_wait\n" \
+ "0:\tmovl %8, %%edx\n" \
+ "2:\tcall __lll_timedlock_wait\n" \
"3:\tjmp 18f\n" \
- "4:\t.size _L_mutex_timedlock_%=, 4b-1b\n\t" \
+ "4:\t.size _L_timedlock_%=, 4b-1b\n\t" \
".previous\n" \
LLL_STUB_UNWIND_INFO_4 \
"18:" \
: "=a" (result), "=c" (ignore1), "=&d" (ignore2), \
- "=m" (futex) \
- : "0" (0), "1" (1), "m" (futex), "m" (timeout) \
+ "=m" (futex), "=S" (ignore3) \
+ : "0" (0), "1" (1), "m" (futex), "m" (timeout), \
+ "4" (private) \
: "memory"); \
result; })
-#define lll_robust_mutex_timedlock(futex, timeout, id) \
- ({ int result, ignore1, ignore2; \
+#define lll_robust_timedlock(futex, timeout, id, private) \
+ ({ int result, ignore1, ignore2, ignore3; \
__asm __volatile (LOCK_INSTR "cmpxchgl %1, %3\n\t" \
- "jnz _L_robust_mutex_timedlock_%=\n\t" \
+ "jnz _L_robust_timedlock_%=\n\t" \
".subsection 1\n\t" \
- ".type _L_robust_mutex_timedlock_%=,@function\n" \
- "_L_robust_mutex_timedlock_%=:\n" \
+ ".type _L_robust_timedlock_%=,@function\n" \
+ "_L_robust_timedlock_%=:\n" \
"1:\tleal %3, %%ecx\n" \
- "0:\tmovl %7, %%edx\n" \
- "2:\tcall __lll_robust_mutex_timedlock_wait\n" \
+ "0:\tmovl %8, %%edx\n" \
+ "2:\tcall __lll_robust_timedlock_wait\n" \
"3:\tjmp 18f\n" \
- "4:\t.size _L_robust_mutex_timedlock_%=, 4b-1b\n\t" \
+ "4:\t.size _L_robust_timedlock_%=, 4b-1b\n\t" \
".previous\n" \
LLL_STUB_UNWIND_INFO_4 \
"18:" \
: "=a" (result), "=c" (ignore1), "=&d" (ignore2), \
- "=m" (futex) \
- : "0" (0), "1" (id), "m" (futex), "m" (timeout) \
+ "=m" (futex), "=S" (ignore3) \
+ : "0" (0), "1" (id), "m" (futex), "m" (timeout), \
+ "4" (private) \
: "memory"); \
result; })
-
-#define lll_mutex_unlock(futex) \
- (void) ({ int ignore; \
- __asm __volatile (LOCK_INSTR "subl $1, %0\n\t" \
- "jne _L_mutex_unlock_%=\n\t" \
- ".subsection 1\n\t" \
- ".type _L_mutex_unlock_%=,@function\n" \
- "_L_mutex_unlock_%=:\n" \
- "1:\tleal %0, %%eax\n" \
- "2:\tcall __lll_mutex_unlock_wake\n" \
- "3:\tjmp 18f\n" \
- "4:\t.size _L_mutex_unlock_%=, 4b-1b\n\t" \
- ".previous\n" \
- LLL_STUB_UNWIND_INFO_3 \
- "18:" \
- : "=m" (futex), "=&a" (ignore) \
- : "m" (futex) \
- : "memory"); })
-
-
-#define lll_robust_mutex_unlock(futex) \
- (void) ({ int ignore; \
- __asm __volatile (LOCK_INSTR "andl %2, %0\n\t" \
- "jne _L_robust_mutex_unlock_%=\n\t" \
- ".subsection 1\n\t" \
- ".type _L_robust_mutex_unlock_%=,@function\n" \
- "_L_robust_mutex_unlock_%=:\n\t" \
- "1:\tleal %0, %%eax\n" \
- "2:\tcall __lll_mutex_unlock_wake\n" \
- "3:\tjmp 18f\n" \
- "4:\t.size _L_robust_mutex_unlock_%=, 4b-1b\n\t"\
- ".previous\n" \
- LLL_STUB_UNWIND_INFO_3 \
- "18:" \
- : "=m" (futex), "=&a" (ignore) \
- : "i" (FUTEX_WAITERS), "m" (futex) \
- : "memory"); })
-
-
-#define lll_robust_mutex_dead(futex) \
- (void) ({ int __ignore; \
- register int _nr asm ("edx") = 1; \
- __asm __volatile (LOCK_INSTR "orl %5, (%2)\n\t" \
- LLL_EBX_LOAD \
- LLL_ENTER_KERNEL \
- LLL_EBX_LOAD \
- : "=a" (__ignore) \
- : "0" (SYS_futex), LLL_EBX_REG (&(futex)), \
- "c" (FUTEX_WAKE), "d" (_nr), \
- "i" (FUTEX_OWNER_DIED), \
- "i" (offsetof (tcbhead_t, sysinfo))); })
-
-
-#define lll_futex_wake(futex, nr) \
- do { \
- int __ignore; \
- register __typeof (nr) _nr asm ("edx") = (nr); \
- __asm __volatile (LLL_EBX_LOAD \
- LLL_ENTER_KERNEL \
- LLL_EBX_LOAD \
- : "=a" (__ignore) \
- : "0" (SYS_futex), LLL_EBX_REG (futex), \
- "c" (FUTEX_WAKE), "d" (_nr), \
- "i" (0) /* phony, to align next arg's number */, \
- "i" (offsetof (tcbhead_t, sysinfo))); \
- } while (0)
-
-
-#define lll_mutex_islocked(futex) \
- (futex != 0)
-
-
-/* We have a separate internal lock implementation which is not tied
- to binary compatibility. */
-
-/* Type for lock object. */
-typedef int lll_lock_t;
-
-/* Initializers for lock. */
-#define LLL_LOCK_INITIALIZER (0)
-#define LLL_LOCK_INITIALIZER_LOCKED (1)
-
-
-extern int __lll_lock_wait (int val, int *__futex)
- __attribute ((regparm (2))) attribute_hidden;
-extern int __lll_unlock_wake (int *__futex)
- __attribute ((regparm (1))) attribute_hidden;
-extern int lll_unlock_wake_cb (int *__futex) attribute_hidden;
-
-
-/* The states of a lock are:
- 0 - untaken
- 1 - taken by one user
- 2 - taken by more users */
-
-
#if defined NOT_IN_libc || defined UP
-# define lll_trylock(futex) lll_mutex_trylock (futex)
-# define lll_lock(futex) lll_mutex_lock (futex)
-# define lll_unlock(futex) lll_mutex_unlock (futex)
+# define __lll_unlock_asm LOCK_INSTR "subl $1, %0\n\t"
#else
-/* Special versions of the macros for use in libc itself. They avoid
- the lock prefix when the thread library is not used.
-
- XXX In future we might even want to avoid it on UP machines. */
-# include <tls.h>
-
-# define lll_trylock(futex) \
- ({ unsigned char ret; \
- __asm __volatile ("cmpl $0, %%gs:%P5\n\t" \
- "je 0f\n\t" \
- "lock\n" \
- "0:\tcmpxchgl %2, %1; setne %0" \
- : "=a" (ret), "=m" (futex) \
- : "r" (LLL_MUTEX_LOCK_INITIALIZER_LOCKED), "m" (futex),\
- "0" (LLL_MUTEX_LOCK_INITIALIZER), \
- "i" (offsetof (tcbhead_t, multiple_threads)) \
- : "memory"); \
- ret; })
-
-
-# define lll_lock(futex) \
- (void) ({ int ignore1, ignore2; \
- __asm __volatile ("cmpl $0, %%gs:%P6\n\t" \
- "je 0f\n\t" \
- "lock\n" \
- "0:\tcmpxchgl %1, %2\n\t" \
- "jnz _L_lock_%=\n\t" \
- ".subsection 1\n\t" \
- ".type _L_lock_%=,@function\n" \
- "_L_lock_%=:\n" \
- "1:\tleal %2, %%ecx\n" \
- "2:\tcall __lll_mutex_lock_wait\n" \
- "3:\tjmp 18f\n" \
- "4:\t.size _L_lock_%=, 4b-1b\n\t" \
- ".previous\n" \
- LLL_STUB_UNWIND_INFO_3 \
- "18:" \
- : "=a" (ignore1), "=c" (ignore2), "=m" (futex) \
- : "0" (0), "1" (1), "m" (futex), \
- "i" (offsetof (tcbhead_t, multiple_threads)) \
- : "memory"); })
-
-
-# define lll_unlock(futex) \
- (void) ({ int ignore; \
- __asm __volatile ("cmpl $0, %%gs:%P3\n\t" \
- "je 0f\n\t" \
- "lock\n" \
- "0:\tsubl $1,%0\n\t" \
- "jne _L_unlock_%=\n\t" \
- ".subsection 1\n\t" \
- ".type _L_unlock_%=,@function\n" \
- "_L_unlock_%=:\n" \
- "1:\tleal %0, %%eax\n" \
- "2:\tcall __lll_mutex_unlock_wake\n" \
- "3:\tjmp 18f\n\t" \
- "4:\t.size _L_unlock_%=, 4b-1b\n\t" \
- ".previous\n" \
- LLL_STUB_UNWIND_INFO_3 \
- "18:" \
- : "=m" (futex), "=&a" (ignore) \
- : "m" (futex), \
- "i" (offsetof (tcbhead_t, multiple_threads)) \
- : "memory"); })
+# define __lll_unlock_asm "cmpl $0, %%gs:%P3\n\t" \
+ "je 0f\n\t" \
+ "lock\n" \
+ "0:\tsubl $1,%0\n\t"
#endif
+#define lll_unlock(futex, private) \
+ (void) \
+ ({ int ignore; \
+ if (__builtin_constant_p (private) && (private) == LLL_PRIVATE) \
+ __asm __volatile (__lll_unlock_asm \
+ "jne _L_unlock_%=\n\t" \
+ ".subsection 1\n\t" \
+ ".type _L_unlock_%=,@function\n" \
+ "_L_unlock_%=:\n" \
+ "1:\tleal %0, %%eax\n" \
+ "2:\tcall __lll_unlock_wake_private\n" \
+ "3:\tjmp 18f\n" \
+ "4:\t.size _L_unlock_%=, 4b-1b\n\t" \
+ ".previous\n" \
+ LLL_STUB_UNWIND_INFO_3 \
+ "18:" \
+ : "=m" (futex), "=&a" (ignore) \
+ : "m" (futex), "i" (MULTIPLE_THREADS_OFFSET) \
+ : "memory"); \
+ else \
+ { \
+ int ignore2; \
+ __asm __volatile (__lll_unlock_asm \
+ "jne _L_unlock_%=\n\t" \
+ ".subsection 1\n\t" \
+ ".type _L_unlock_%=,@function\n" \
+ "_L_unlock_%=:\n" \
+ "1:\tleal %0, %%eax\n" \
+ "0:\tmovl %5, %%ecx\n" \
+ "2:\tcall __lll_unlock_wake\n" \
+ "3:\tjmp 18f\n" \
+ "4:\t.size _L_unlock_%=, 4b-1b\n\t" \
+ ".previous\n" \
+ LLL_STUB_UNWIND_INFO_4 \
+ "18:" \
+ : "=m" (futex), "=&a" (ignore), "=&c" (ignore2) \
+ : "i" (MULTIPLE_THREADS_OFFSET), "m" (futex), \
+ "g" (private) \
+ : "memory"); \
+ } \
+ })
+
+#define lll_robust_unlock(futex, private) \
+ (void) \
+ ({ int ignore, ignore2; \
+ __asm __volatile (LOCK_INSTR "andl %3, %0\n\t" \
+ "jne _L_robust_unlock_%=\n\t" \
+ ".subsection 1\n\t" \
+ ".type _L_robust_unlock_%=,@function\n" \
+ "_L_robust_unlock_%=:\n\t" \
+ "1:\tleal %0, %%eax\n" \
+ "0:\tmovl %5, %%ecx\n" \
+ "2:\tcall __lll_unlock_wake\n" \
+ "3:\tjmp 18f\n" \
+ "4:\t.size _L_robust_unlock_%=, 4b-1b\n\t" \
+ ".previous\n" \
+ LLL_STUB_UNWIND_INFO_4 \
+ "18:" \
+ : "=m" (futex), "=&a" (ignore), "=&c" (ignore2) \
+ : "i" (FUTEX_WAITERS), "m" (futex), "g" (private) \
+ : "memory"); \
+ })
+
+
+#define lll_robust_dead(futex, private) \
+ (void) \
+ ({ int __ignore; \
+ register int _nr asm ("edx") = 1; \
+ __asm __volatile (LOCK_INSTR "orl %5, (%2)\n\t" \
+ LLL_EBX_LOAD \
+ LLL_ENTER_KERNEL \
+ LLL_EBX_LOAD \
+ : "=a" (__ignore) \
+ : "0" (SYS_futex), LLL_EBX_REG (&(futex)), \
+ "c" (__lll_private_flag (FUTEX_WAKE, private)), \
+ "d" (_nr), "i" (FUTEX_OWNER_DIED), \
+ "i" (offsetof (tcbhead_t, sysinfo))); \
+ })
#define lll_islocked(futex) \
(futex != LLL_LOCK_INITIALIZER)
-
/* The kernel notifies a process with uses CLONE_CLEARTID via futex
wakeup when the clone terminates. The memory location contains the
thread ID while the clone is running and is reset to zero
@@ -571,28 +572,6 @@ extern int __lll_timedwait_tid (int *tid
} \
__result; })
-
-/* Conditional variable handling. */
-
-extern void __lll_cond_wait (pthread_cond_t *cond)
- __attribute ((regparm (1))) attribute_hidden;
-extern int __lll_cond_timedwait (pthread_cond_t *cond,
- const struct timespec *abstime)
- __attribute ((regparm (2))) attribute_hidden;
-extern void __lll_cond_wake (pthread_cond_t *cond)
- __attribute ((regparm (1))) attribute_hidden;
-extern void __lll_cond_broadcast (pthread_cond_t *cond)
- __attribute ((regparm (1))) attribute_hidden;
-
-
-#define lll_cond_wait(cond) \
- __lll_cond_wait (cond)
-#define lll_cond_timedwait(cond, abstime) \
- __lll_cond_timedwait (cond, abstime)
-#define lll_cond_wake(cond) \
- __lll_cond_wake (cond)
-#define lll_cond_broadcast(cond) \
- __lll_cond_broadcast (cond)
-
+#endif /* !__ASSEMBLER__ */
#endif /* lowlevellock.h */
--- libc/nptl/sysdeps/unix/sysv/linux/i386/pthread_once.S 3 Oct 2003 19:49:23 -0000 1.12
+++ libc/nptl/sysdeps/unix/sysv/linux/i386/pthread_once.S 1 Aug 2007 04:23:14 -0000 1.14
@@ -19,15 +19,9 @@
#include <unwindbuf.h>
#include <sysdep.h>
+#include <kernel-features.h>
+#include <lowlevellock.h>
-#ifndef UP
-# define LOCK lock
-#else
-# define LOCK
-#endif
-
-#define SYS_futex 240
-#define FUTEX_WAKE 1
.comm __fork_generation, 4, 4
@@ -90,7 +84,16 @@ __pthread_once:
jnz 3f /* Different for generation -> run initializer. */
/* Somebody else got here first. Wait. */
- movl %esi, %ecx /* movl $FUTEX_WAIT, %ecx */
+#ifdef __ASSUME_PRIVATE_FUTEX
+ movl $FUTEX_WAIT|FUTEX_PRIVATE_FLAG, %ecx
+#else
+# if FUTEX_WAIT == 0
+ movl %gs:PRIVATE_FUTEX, %ecx
+# else
+ movl $FUTEX_WAIT, %ecx
+ orl %gs:PRIVATE_FUTEX, %ecx
+# endif
+#endif
movl $SYS_futex, %eax
ENTER_KERNEL
jmp 6b
@@ -131,7 +134,12 @@ __pthread_once:
/* Wake up all other threads. */
movl $0x7fffffff, %edx
+#ifdef __ASSUME_PRIVATE_FUTEX
+ movl $FUTEX_WAKE|FUTEX_PRIVATE_FLAG, %ecx
+#else
movl $FUTEX_WAKE, %ecx
+ orl %gs:PRIVATE_FUTEX, %ecx
+#endif
movl $SYS_futex, %eax
ENTER_KERNEL
@@ -152,7 +160,12 @@ __pthread_once:
movl $0, (%ebx)
movl $0x7fffffff, %edx
+#ifdef __ASSUME_PRIVATE_FUTEX
+ movl $FUTEX_WAKE|FUTEX_PRIVATE_FLAG, %ecx
+#else
movl $FUTEX_WAKE, %ecx
+ orl %gs:PRIVATE_FUTEX, %ecx
+#endif
movl $SYS_futex, %eax
ENTER_KERNEL
--- libc/nptl/sysdeps/unix/sysv/linux/i386/bits/pthreadtypes.h 13 Feb 2006 01:26:15 -0000 1.23
+++ libc/nptl/sysdeps/unix/sysv/linux/i386/bits/pthreadtypes.h 27 May 2007 18:00:12 -0000 1.24
@@ -128,7 +128,10 @@ typedef union
unsigned int __nr_writers_queued;
/* FLAGS must stay at this position in the structure to maintain
binary compatibility. */
- unsigned int __flags;
+ unsigned char __flags;
+ unsigned char __shared;
+ unsigned char __pad1;
+ unsigned char __pad2;
int __writer;
} __data;
char __size[__SIZEOF_PTHREAD_RWLOCK_T];
--- libc/nptl/sysdeps/unix/sysv/linux/i386/i486/libc-lowlevellock.S 9 Apr 2006 02:42:29 -0000 1.12
+++ libc/nptl/sysdeps/unix/sysv/linux/i386/i486/libc-lowlevellock.S 1 Aug 2007 04:28:38 -0000 1.15
@@ -17,14 +17,4 @@
Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
02111-1307 USA. */
-/* In libc.so we do not unconditionally use the lock prefix. Only if
- the application is using threads. */
-#ifndef UP
-# define LOCK \
- cmpl $0, %gs:MULTIPLE_THREADS_OFFSET; \
- je 0f; \
- lock; \
-0:
-#endif
-
#include "lowlevellock.S"
--- libc/nptl/sysdeps/unix/sysv/linux/i386/i486/lowlevellock.S 5 Sep 2006 14:46:43 -0000 1.16
+++ libc/nptl/sysdeps/unix/sysv/linux/i386/i486/lowlevellock.S 1 Aug 2007 04:26:13 -0000 1.19
@@ -19,28 +19,53 @@
#include <sysdep.h>
#include <pthread-errnos.h>
+#include <kernel-features.h>
+#include <lowlevellock.h>
.text
-#ifndef LOCK
-# ifdef UP
-# define LOCK
+#ifdef __ASSUME_PRIVATE_FUTEX
+# define LOAD_PRIVATE_FUTEX_WAIT(reg) \
+ movl $(FUTEX_WAIT | FUTEX_PRIVATE_FLAG), reg
+# define LOAD_PRIVATE_FUTEX_WAKE(reg) \
+ movl $(FUTEX_WAKE | FUTEX_PRIVATE_FLAG), reg
+# define LOAD_FUTEX_WAIT(reg) \
+ xorl $(FUTEX_WAIT | FUTEX_PRIVATE_FLAG), reg
+# define LOAD_FUTEX_WAKE(reg) \
+ xorl $(FUTEX_WAKE | FUTEX_PRIVATE_FLAG), reg
+#else
+# if FUTEX_WAIT == 0
+# define LOAD_PRIVATE_FUTEX_WAIT(reg) \
+ movl %gs:PRIVATE_FUTEX, reg
# else
-# define LOCK lock
+# define LOAD_PRIVATE_FUTEX_WAIT(reg) \
+ movl %gs:PRIVATE_FUTEX, reg ; \
+ orl $FUTEX_WAIT, reg
# endif
+# define LOAD_PRIVATE_FUTEX_WAKE(reg) \
+ movl %gs:PRIVATE_FUTEX, reg ; \
+ orl $FUTEX_WAKE, reg
+# if FUTEX_WAIT == 0
+# define LOAD_FUTEX_WAIT(reg) \
+ xorl $FUTEX_PRIVATE_FLAG, reg ; \
+ andl %gs:PRIVATE_FUTEX, reg
+# else
+# define LOAD_FUTEX_WAIT(reg) \
+ xorl $FUTEX_PRIVATE_FLAG, reg ; \
+ andl %gs:PRIVATE_FUTEX, reg ; \
+ orl $FUTEX_WAIT, reg
+# endif
+# define LOAD_FUTEX_WAKE(reg) \
+ xorl $FUTEX_PRIVATE_FLAG, reg ; \
+ andl %gs:PRIVATE_FUTEX, reg ; \
+ orl $FUTEX_WAKE, reg
#endif
-#define SYS_gettimeofday __NR_gettimeofday
-#define SYS_futex 240
-#define FUTEX_WAIT 0
-#define FUTEX_WAKE 1
-
-
- .globl __lll_mutex_lock_wait
- .type __lll_mutex_lock_wait,@function
- .hidden __lll_mutex_lock_wait
+ .globl __lll_lock_wait_private
+ .type __lll_lock_wait_private,@function
+ .hidden __lll_lock_wait_private
.align 16
-__lll_mutex_lock_wait:
+__lll_lock_wait_private:
cfi_startproc
pushl %edx
cfi_adjust_cfa_offset(4)
@@ -55,7 +80,7 @@ __lll_mutex_lock_wait:
movl $2, %edx
movl %ecx, %ebx
xorl %esi, %esi /* No timeout. */
- xorl %ecx, %ecx /* movl $FUTEX_WAIT, %ecx */
+ LOAD_PRIVATE_FUTEX_WAIT (%ecx)
cmpl %edx, %eax /* NB: %edx == 2 */
jne 2f
@@ -80,15 +105,60 @@ __lll_mutex_lock_wait:
cfi_restore(%edx)
ret
cfi_endproc
- .size __lll_mutex_lock_wait,.-__lll_mutex_lock_wait
-
+ .size __lll_lock_wait_private,.-__lll_lock_wait_private
#ifdef NOT_IN_libc
- .globl __lll_mutex_timedlock_wait
- .type __lll_mutex_timedlock_wait,@function
- .hidden __lll_mutex_timedlock_wait
+ .globl __lll_lock_wait
+ .type __lll_lock_wait,@function
+ .hidden __lll_lock_wait
+ .align 16
+__lll_lock_wait:
+ cfi_startproc
+ pushl %edx
+ cfi_adjust_cfa_offset(4)
+ pushl %ebx
+ cfi_adjust_cfa_offset(4)
+ pushl %esi
+ cfi_adjust_cfa_offset(4)
+ cfi_offset(%edx, -8)
+ cfi_offset(%ebx, -12)
+ cfi_offset(%esi, -16)
+
+ movl %edx, %ebx
+ movl $2, %edx
+ xorl %esi, %esi /* No timeout. */
+ LOAD_FUTEX_WAIT (%ecx)
+
+ cmpl %edx, %eax /* NB: %edx == 2 */
+ jne 2f
+
+1: movl $SYS_futex, %eax
+ ENTER_KERNEL
+
+2: movl %edx, %eax
+ xchgl %eax, (%ebx) /* NB: lock is implied */
+
+ testl %eax, %eax
+ jnz 1b
+
+ popl %esi
+ cfi_adjust_cfa_offset(-4)
+ cfi_restore(%esi)
+ popl %ebx
+ cfi_adjust_cfa_offset(-4)
+ cfi_restore(%ebx)
+ popl %edx
+ cfi_adjust_cfa_offset(-4)
+ cfi_restore(%edx)
+ ret
+ cfi_endproc
+ .size __lll_lock_wait,.-__lll_lock_wait
+
+ .globl __lll_timedlock_wait
+ .type __lll_timedlock_wait,@function
+ .hidden __lll_timedlock_wait
.align 16
-__lll_mutex_timedlock_wait:
+__lll_timedlock_wait:
cfi_startproc
/* Check for a valid timeout value. */
cmpl $1000000000, 4(%edx)
@@ -118,7 +188,7 @@ __lll_mutex_timedlock_wait:
/* Get current time. */
movl %esp, %ebx
xorl %ecx, %ecx
- movl $SYS_gettimeofday, %eax
+ movl $__NR_gettimeofday, %eax
ENTER_KERNEL
/* Compute relative timeout. */
@@ -151,7 +221,8 @@ __lll_mutex_timedlock_wait:
/* Futex call. */
movl %esp, %esi
- xorl %ecx, %ecx /* movl $FUTEX_WAIT, %ecx */
+ movl 16(%esp), %ecx
+ LOAD_FUTEX_WAIT (%ecx)
movl $SYS_futex, %eax
ENTER_KERNEL
movl %eax, %ecx
@@ -201,44 +272,51 @@ __lll_mutex_timedlock_wait:
5: movl $ETIMEDOUT, %eax
jmp 6b
cfi_endproc
- .size __lll_mutex_timedlock_wait,.-__lll_mutex_timedlock_wait
+ .size __lll_timedlock_wait,.-__lll_timedlock_wait
#endif
-
-#ifdef NOT_IN_libc
- .globl lll_unlock_wake_cb
- .type lll_unlock_wake_cb,@function
- .hidden lll_unlock_wake_cb
+ .globl __lll_unlock_wake_private
+ .type __lll_unlock_wake_private,@function
+ .hidden __lll_unlock_wake_private
.align 16
-lll_unlock_wake_cb:
+__lll_unlock_wake_private:
+ cfi_startproc
pushl %ebx
+ cfi_adjust_cfa_offset(4)
pushl %ecx
+ cfi_adjust_cfa_offset(4)
pushl %edx
+ cfi_adjust_cfa_offset(4)
+ cfi_offset(%ebx, -8)
+ cfi_offset(%ecx, -12)
+ cfi_offset(%edx, -16)
- movl 20(%esp), %ebx
- LOCK
- subl $1, (%ebx)
- je 1f
-
- movl $FUTEX_WAKE, %ecx
+ movl %eax, %ebx
+ movl $0, (%eax)
+ LOAD_PRIVATE_FUTEX_WAKE (%ecx)
movl $1, %edx /* Wake one thread. */
movl $SYS_futex, %eax
- movl $0, (%ebx)
ENTER_KERNEL
-1: popl %edx
+ popl %edx
+ cfi_adjust_cfa_offset(-4)
+ cfi_restore(%edx)
popl %ecx
+ cfi_adjust_cfa_offset(-4)
+ cfi_restore(%ecx)
popl %ebx
+ cfi_adjust_cfa_offset(-4)
+ cfi_restore(%ebx)
ret
- .size lll_unlock_wake_cb,.-lll_unlock_wake_cb
-#endif
-
+ cfi_endproc
+ .size __lll_unlock_wake_private,.-__lll_unlock_wake_private
- .globl __lll_mutex_unlock_wake
- .type __lll_mutex_unlock_wake,@function
- .hidden __lll_mutex_unlock_wake
+#ifdef NOT_IN_libc
+ .globl __lll_unlock_wake
+ .type __lll_unlock_wake,@function
+ .hidden __lll_unlock_wake
.align 16
-__lll_mutex_unlock_wake:
+__lll_unlock_wake:
cfi_startproc
pushl %ebx
cfi_adjust_cfa_offset(4)
@@ -252,7 +330,7 @@ __lll_mutex_unlock_wake:
movl %eax, %ebx
movl $0, (%eax)
- movl $FUTEX_WAKE, %ecx
+ LOAD_FUTEX_WAKE (%ecx)
movl $1, %edx /* Wake one thread. */
movl $SYS_futex, %eax
ENTER_KERNEL
@@ -268,10 +346,8 @@ __lll_mutex_unlock_wake:
cfi_restore(%ebx)
ret
cfi_endproc
- .size __lll_mutex_unlock_wake,.-__lll_mutex_unlock_wake
+ .size __lll_unlock_wake,.-__lll_unlock_wake
-
-#ifdef NOT_IN_libc
.globl __lll_timedwait_tid
.type __lll_timedwait_tid,@function
.hidden __lll_timedwait_tid
@@ -289,7 +365,7 @@ __lll_timedwait_tid:
/* Get current time. */
2: movl %esp, %ebx
xorl %ecx, %ecx
- movl $SYS_gettimeofday, %eax
+ movl $__NR_gettimeofday, %eax
ENTER_KERNEL
/* Compute relative timeout. */
@@ -314,6 +390,8 @@ __lll_timedwait_tid:
jz 4f
movl %esp, %esi
+ /* XXX The kernel so far uses global futex for the wakeup at
+ all times. */
xorl %ecx, %ecx /* movl $FUTEX_WAIT, %ecx */
movl %ebp, %ebx
movl $SYS_futex, %eax
--- libc/nptl/sysdeps/unix/sysv/linux/i386/i486/lowlevelrobustlock.S 5 Sep 2006 14:46:43 -0000 1.3
+++ libc/nptl/sysdeps/unix/sysv/linux/i386/i486/lowlevelrobustlock.S 1 Aug 2007 04:24:56 -0000 1.4
@@ -19,31 +19,36 @@
#include <sysdep.h>
#include <pthread-errnos.h>
+#include <lowlevellock.h>
#include <lowlevelrobustlock.h>
+#include <kernel-features.h>
.text
-#ifndef LOCK
-# ifdef UP
-# define LOCK
-# else
-# define LOCK lock
-# endif
-#endif
-
-#define SYS_gettimeofday __NR_gettimeofday
-#define SYS_futex 240
-#define FUTEX_WAIT 0
-#define FUTEX_WAKE 1
#define FUTEX_WAITERS 0x80000000
#define FUTEX_OWNER_DIED 0x40000000
+#ifdef __ASSUME_PRIVATE_FUTEX
+# define LOAD_FUTEX_WAIT(reg) \
+ xorl $(FUTEX_WAIT | FUTEX_PRIVATE_FLAG), reg
+#else
+# if FUTEX_WAIT == 0
+# define LOAD_FUTEX_WAIT(reg) \
+ xorl $FUTEX_PRIVATE_FLAG, reg ; \
+ andl %gs:PRIVATE_FUTEX, reg
+# else
+# define LOAD_FUTEX_WAIT(reg) \
+ xorl $FUTEX_PRIVATE_FLAG, reg ; \
+ andl %gs:PRIVATE_FUTEX, reg ; \
+ orl $FUTEX_WAIT, reg
+# endif
+#endif
- .globl __lll_robust_mutex_lock_wait
- .type __lll_robust_mutex_lock_wait,@function
- .hidden __lll_robust_mutex_lock_wait
+ .globl __lll_robust_lock_wait
+ .type __lll_robust_lock_wait,@function
+ .hidden __lll_robust_lock_wait
.align 16
-__lll_robust_mutex_lock_wait:
+__lll_robust_lock_wait:
cfi_startproc
pushl %edx
cfi_adjust_cfa_offset(4)
@@ -55,9 +60,9 @@ __lll_robust_mutex_lock_wait:
cfi_offset(%ebx, -12)
cfi_offset(%esi, -16)
- movl %ecx, %ebx
+ movl %edx, %ebx
xorl %esi, %esi /* No timeout. */
- xorl %ecx, %ecx /* movl $FUTEX_WAIT, %ecx */
+ LOAD_FUTEX_WAIT (%ecx)
4: movl %eax, %edx
orl $FUTEX_WAITERS, %edx
@@ -98,14 +103,14 @@ __lll_robust_mutex_lock_wait:
cfi_restore(%edx)
ret
cfi_endproc
- .size __lll_robust_mutex_lock_wait,.-__lll_robust_mutex_lock_wait
+ .size __lll_robust_lock_wait,.-__lll_robust_lock_wait
- .globl __lll_robust_mutex_timedlock_wait
- .type __lll_robust_mutex_timedlock_wait,@function
- .hidden __lll_robust_mutex_timedlock_wait
+ .globl __lll_robust_timedlock_wait
+ .type __lll_robust_timedlock_wait,@function
+ .hidden __lll_robust_timedlock_wait
.align 16
-__lll_robust_mutex_timedlock_wait:
+__lll_robust_timedlock_wait:
cfi_startproc
/* Check for a valid timeout value. */
cmpl $1000000000, 4(%edx)
@@ -136,7 +141,7 @@ __lll_robust_mutex_timedlock_wait:
/* Get current time. */
movl %esp, %ebx
xorl %ecx, %ecx
- movl $SYS_gettimeofday, %eax
+ movl $__NR_gettimeofday, %eax
ENTER_KERNEL
/* Compute relative timeout. */
@@ -177,7 +182,8 @@ __lll_robust_mutex_timedlock_wait:
2:
/* Futex call. */
movl %esp, %esi
- xorl %ecx, %ecx /* movl $FUTEX_WAIT, %ecx */
+ movl 20(%esp), %ecx
+ LOAD_FUTEX_WAIT (%ecx)
movl $SYS_futex, %eax
ENTER_KERNEL
movl %eax, %ecx
@@ -224,4 +230,4 @@ __lll_robust_mutex_timedlock_wait:
8: movl $ETIMEDOUT, %eax
jmp 6b
cfi_endproc
- .size __lll_robust_mutex_timedlock_wait,.-__lll_robust_mutex_timedlock_wait
+ .size __lll_robust_timedlock_wait,.-__lll_robust_timedlock_wait
--- libc/nptl/sysdeps/unix/sysv/linux/i386/i486/pthread_barrier_wait.S 9 Apr 2006 02:42:29 -0000 1.12
+++ libc/nptl/sysdeps/unix/sysv/linux/i386/i486/pthread_barrier_wait.S 1 Aug 2007 04:30:20 -0000 1.14
@@ -18,19 +18,9 @@
02111-1307 USA. */
#include <sysdep.h>
+#include <lowlevellock.h>
#include <lowlevelbarrier.h>
-#define SYS_futex 240
-#define FUTEX_WAIT 0
-#define FUTEX_WAKE 1
-
-#ifndef UP
-# define LOCK lock
-#else
-# define LOCK
-#endif
-
-
.text
.globl pthread_barrier_wait
@@ -69,7 +59,13 @@ pthread_barrier_wait:
/* Wait for the remaining threads. The call will return immediately
if the CURR_EVENT memory has meanwhile been changed. */
-7: xorl %ecx, %ecx /* movl $FUTEX_WAIT, %ecx */
+7:
+#if FUTEX_WAIT == 0
+ movl PRIVATE(%ebx), %ecx
+#else
+ movl $FUTEX_WAIT, %ecx
+ orl PRIVATE(%ebx), %ecx
+#endif
xorl %esi, %esi
8: movl $SYS_futex, %eax
ENTER_KERNEL
@@ -120,6 +116,7 @@ pthread_barrier_wait:
so 0x7fffffff is the highest value. */
movl $0x7fffffff, %edx
movl $FUTEX_WAKE, %ecx
+ orl PRIVATE(%ebx), %ecx
movl $SYS_futex, %eax
ENTER_KERNEL
@@ -145,19 +142,27 @@ pthread_barrier_wait:
popl %ebx
ret
-1: leal MUTEX(%ebx), %ecx
- call __lll_mutex_lock_wait
+1: movl PRIVATE(%ebx), %ecx
+ leal MUTEX(%ebx), %edx
+ xorl $LLL_SHARED, %ecx
+ call __lll_lock_wait
jmp 2b
-4: leal MUTEX(%ebx), %eax
- call __lll_mutex_unlock_wake
+4: movl PRIVATE(%ebx), %ecx
+ leal MUTEX(%ebx), %eax
+ xorl $LLL_SHARED, %ecx
+ call __lll_unlock_wake
jmp 5b
-6: leal MUTEX(%ebx), %eax
- call __lll_mutex_unlock_wake
+6: movl PRIVATE(%ebx), %ecx
+ leal MUTEX(%ebx), %eax
+ xorl $LLL_SHARED, %ecx
+ call __lll_unlock_wake
jmp 7b
-9: leal MUTEX(%ebx), %eax
- call __lll_mutex_unlock_wake
+9: movl PRIVATE(%ebx), %ecx
+ leal MUTEX(%ebx), %eax
+ xorl $LLL_SHARED, %ecx
+ call __lll_unlock_wake
jmp 10b
.size pthread_barrier_wait,.-pthread_barrier_wait
--- libc/nptl/sysdeps/unix/sysv/linux/i386/i486/pthread_cond_broadcast.S 29 Jul 2006 04:31:49 -0000 1.14
+++ libc/nptl/sysdeps/unix/sysv/linux/i386/i486/pthread_cond_broadcast.S 14 Aug 2007 02:26:09 -0000 1.16
@@ -19,24 +19,11 @@
#include <sysdep.h>
#include <shlib-compat.h>
+#include <lowlevellock.h>
#include <lowlevelcond.h>
#include <kernel-features.h>
#include <pthread-pi-defines.h>
-
-#ifdef UP
-# define LOCK
-#else
-# define LOCK lock
-#endif
-
-#define SYS_futex 240
-#define FUTEX_WAIT 0
-#define FUTEX_WAKE 1
-#define FUTEX_REQUEUE 3
-#define FUTEX_CMP_REQUEUE 4
-
-#define EINVAL 22
-
+#include <pthread-errnos.h>
.text
@@ -96,11 +83,18 @@ __pthread_cond_broadcast:
je 9f
/* XXX: The kernel so far doesn't support requeue to PI futex. */
- testl $PI_BIT, MUTEX_KIND(%edi)
+ /* XXX: The kernel only supports FUTEX_CMP_REQUEUE to the same
+ type of futex (private resp. shared). */
+ testl $(PI_BIT | PS_BIT), MUTEX_KIND(%edi)
jne 9f
/* Wake up all threads. */
- movl $FUTEX_CMP_REQUEUE, %ecx
+#ifdef __ASSUME_PRIVATE_FUTEX
+ movl $(FUTEX_CMP_REQUEUE|FUTEX_PRIVATE_FLAG), %ecx
+#else
+ movl %gs:PRIVATE_FUTEX, %ecx
+ orl $FUTEX_CMP_REQUEUE, %ecx
+#endif
movl $SYS_futex, %eax
movl $0x7fffffff, %esi
movl $1, %edx
@@ -141,26 +135,67 @@ __pthread_cond_broadcast:
/* Initial locking failed. */
1:
#if cond_lock == 0
- movl %ebx, %ecx
+ movl %ebx, %edx
#else
- leal cond_lock(%ebx), %ecx
+ leal cond_lock(%ebx), %edx
+#endif
+#if (LLL_SHARED-LLL_PRIVATE) > 255
+ xorl %ecx, %ecx
#endif
- call __lll_mutex_lock_wait
+ cmpl $-1, dep_mutex(%ebx)
+ setne %cl
+ subl $1, %ecx
+ andl $(LLL_SHARED-LLL_PRIVATE), %ecx
+#if LLL_PRIVATE != 0
+ addl $LLL_PRIVATE, %ecx
+#endif
+ call __lll_lock_wait
jmp 2b
/* Unlock in loop requires waekup. */
5: leal cond_lock-cond_futex(%ebx), %eax
- call __lll_mutex_unlock_wake
+#if (LLL_SHARED-LLL_PRIVATE) > 255
+ xorl %ecx, %ecx
+#endif
+ cmpl $-1, dep_mutex-cond_futex(%ebx)
+ setne %cl
+ subl $1, %ecx
+ andl $(LLL_SHARED-LLL_PRIVATE), %ecx
+#if LLL_PRIVATE != 0
+ addl $LLL_PRIVATE, %ecx
+#endif
+ call __lll_unlock_wake
jmp 6b
/* Unlock in loop requires waekup. */
7: leal cond_lock-cond_futex(%ebx), %eax
- call __lll_mutex_unlock_wake
+#if (LLL_SHARED-LLL_PRIVATE) > 255
+ xorl %ecx, %ecx
+#endif
+ cmpl $-1, dep_mutex-cond_futex(%ebx)
+ setne %cl
+ subl $1, %ecx
+ andl $(LLL_SHARED-LLL_PRIVATE), %ecx
+#if LLL_PRIVATE != 0
+ addl $LLL_PRIVATE, %ecx
+#endif
+ call __lll_unlock_wake
jmp 8b
9: /* The futex requeue functionality is not available. */
movl $0x7fffffff, %edx
- movl $FUTEX_WAKE, %ecx
+#if FUTEX_PRIVATE_FLAG > 255
+ xorl %ecx, %ecx
+#endif
+ cmpl $-1, dep_mutex-cond_futex(%ebx)
+ sete %cl
+ subl $1, %ecx
+#ifdef __ASSUME_PRIVATE_FUTEX
+ andl $FUTEX_PRIVATE_FLAG, %ecx
+#else
+ andl %gs:PRIVATE_FUTEX, %ecx
+#endif
+ addl $FUTEX_WAKE, %ecx
movl $SYS_futex, %eax
ENTER_KERNEL
jmp 10b
--- libc/nptl/sysdeps/unix/sysv/linux/i386/i486/pthread_cond_signal.S 8 Sep 2005 17:40:52 -0000 1.15
+++ libc/nptl/sysdeps/unix/sysv/linux/i386/i486/pthread_cond_signal.S 14 Aug 2007 02:26:35 -0000 1.17
@@ -19,23 +19,10 @@
#include <sysdep.h>
#include <shlib-compat.h>
+#include <lowlevellock.h>
#include <lowlevelcond.h>
#include <kernel-features.h>
-
-#ifdef UP
-# define LOCK
-#else
-# define LOCK lock
-#endif
-
-#define SYS_futex 240
-#define FUTEX_WAIT 0
-#define FUTEX_WAKE 1
-#define FUTEX_WAKE_OP 5
-
-#define FUTEX_OP_CLEAR_WAKE_IF_GT_ONE ((4 << 24) | 1)
-
-#define EINVAL 22
+#include <pthread-errnos.h>
.text
@@ -83,7 +70,18 @@ __pthread_cond_signal:
/* Wake up one thread. */
pushl %esi
pushl %ebp
- movl $FUTEX_WAKE_OP, %ecx
+#if FUTEX_PRIVATE_FLAG > 255
+ xorl %ecx, %ecx
+#endif
+ cmpl $-1, dep_mutex-cond_futex(%ebx)
+ sete %cl
+ subl $1, %ecx
+#ifdef __ASSUME_PRIVATE_FUTEX
+ andl $FUTEX_PRIVATE_FLAG, %ecx
+#else
+ andl %gs:PRIVATE_FUTEX, %ecx
+#endif
+ addl $FUTEX_WAKE_OP, %ecx
movl $SYS_futex, %eax
movl $1, %edx
movl $1, %esi
@@ -105,7 +103,9 @@ __pthread_cond_signal:
popl %ebx
ret
-7: movl $FUTEX_WAKE, %ecx
+7: /* %ecx should be either FUTEX_WAKE_OP or
+ FUTEX_WAKE_OP|FUTEX_PRIVATE_FLAG from the previous syscall. */
+ xorl $(FUTEX_WAKE ^ FUTEX_WAKE_OP), %ecx
movl $SYS_futex, %eax
/* %edx should be 1 already from $FUTEX_WAKE_OP syscall.
movl $1, %edx */
@@ -119,17 +119,37 @@ __pthread_cond_signal:
/* Unlock in loop requires wakeup. */
5: movl %edi, %eax
- call __lll_mutex_unlock_wake
+#if (LLL_SHARED-LLL_PRIVATE) > 255
+ xorl %ecx, %ecx
+#endif
+ cmpl $-1, dep_mutex-cond_futex(%ebx)
+ setne %cl
+ subl $1, %ecx
+ andl $(LLL_SHARED-LLL_PRIVATE), %ecx
+#if LLL_PRIVATE != 0
+ addl $LLL_PRIVATE, %ecx
+#endif
+ call __lll_unlock_wake
jmp 6b
/* Initial locking failed. */
1:
#if cond_lock == 0
- movl %edi, %ecx
+ movl %edi, %edx
#else
- leal cond_lock(%edi), %ecx
+ leal cond_lock(%edi), %edx
+#endif
+#if (LLL_SHARED-LLL_PRIVATE) > 255
+ xorl %ecx, %ecx
+#endif
+ cmpl $-1, dep_mutex(%edi)
+ setne %cl
+ subl $1, %ecx
+ andl $(LLL_SHARED-LLL_PRIVATE), %ecx
+#if LLL_PRIVATE != 0
+ addl $LLL_PRIVATE, %ecx
#endif
- call __lll_mutex_lock_wait
+ call __lll_lock_wait
jmp 2b
.size __pthread_cond_signal, .-__pthread_cond_signal
--- libc/nptl/sysdeps/unix/sysv/linux/i386/i486/pthread_cond_timedwait.S 9 Sep 2006 11:21:23 -0000 1.39
+++ libc/nptl/sysdeps/unix/sysv/linux/i386/i486/pthread_cond_timedwait.S 14 Aug 2007 02:26:51 -0000 1.43
@@ -19,19 +19,10 @@
#include <sysdep.h>
#include <shlib-compat.h>
+#include <lowlevellock.h>
#include <lowlevelcond.h>
#include <pthread-errnos.h>
-
-#ifdef UP
-# define LOCK
-#else
-# define LOCK lock
-#endif
-
-#define SYS_gettimeofday __NR_gettimeofday
-#define SYS_futex 240
-#define FUTEX_WAIT 0
-#define FUTEX_WAKE 1
+#include <kernel-features.h>
.text
@@ -87,7 +78,7 @@ __pthread_cond_timedwait:
addl $1, total_seq(%ebx)
adcl $0, total_seq+4(%ebx)
addl $1, cond_futex(%ebx)
- addl $(1 << clock_bits), cond_nwaiters(%ebx)
+ addl $(1 << nwaiters_shift), cond_nwaiters(%ebx)
#define FRAME_SIZE 24
subl $FRAME_SIZE, %esp
@@ -106,7 +97,7 @@ __pthread_cond_timedwait:
#ifdef __NR_clock_gettime
/* Get the clock number. */
movl cond_nwaiters(%ebx), %ebx
- andl $((1 << clock_bits) - 1), %ebx
+ andl $((1 << nwaiters_shift) - 1), %ebx
/* Only clocks 0 and 1 are allowed so far. Both are handled in the
kernel. */
leal 4(%esp), %ecx
@@ -127,7 +118,7 @@ __pthread_cond_timedwait:
/* Get the current time. */
leal 4(%esp), %ebx
xorl %ecx, %ecx
- movl $SYS_gettimeofday, %eax
+ movl $__NR_gettimeofday, %eax
ENTER_KERNEL
movl %edx, %ebx
@@ -167,7 +158,20 @@ __pthread_cond_timedwait:
movl %eax, (%esp)
leal 4(%esp), %esi
- xorl %ecx, %ecx /* movl $FUTEX_WAIT, %ecx */
+#if FUTEX_PRIVATE_FLAG > 255
+ xorl %ecx, %ecx
+#endif
+ cmpl $-1, dep_mutex(%ebx)
+ sete %cl
+ subl $1, %ecx
+#ifdef __ASSUME_PRIVATE_FUTEX
+ andl $FUTEX_PRIVATE_FLAG, %ecx
+#else
+ andl %gs:PRIVATE_FUTEX, %ecx
+#endif
+#if FUTEX_WAIT != 0
+ addl $FUTEX_WAIT, %ecx
+#endif
movl %edi, %edx
addl $cond_futex, %ebx
.Ladd_cond_futex:
@@ -228,7 +232,7 @@ __pthread_cond_timedwait:
14: addl $1, woken_seq(%ebx)
adcl $0, woken_seq+4(%ebx)
-24: subl $(1 << clock_bits), cond_nwaiters(%ebx)
+24: subl $(1 << nwaiters_shift), cond_nwaiters(%ebx)
/* Wake up a thread which wants to destroy the condvar object. */
movl total_seq(%ebx), %eax
@@ -236,12 +240,23 @@ __pthread_cond_timedwait:
cmpl $0xffffffff, %eax
jne 25f
movl cond_nwaiters(%ebx), %eax
- andl $~((1 << clock_bits) - 1), %eax
+ andl $~((1 << nwaiters_shift) - 1), %eax
jne 25f
addl $cond_nwaiters, %ebx
movl $SYS_futex, %eax
- movl $FUTEX_WAKE, %ecx
+#if FUTEX_PRIVATE_FLAG > 255
+ xorl %ecx, %ecx
+#endif
+ cmpl $-1, dep_mutex-cond_nwaiters(%ebx)
+ sete %cl
+ subl $1, %ecx
+#ifdef __ASSUME_PRIVATE_FUTEX
+ andl $FUTEX_PRIVATE_FLAG, %ecx
+#else
+ andl %gs:PRIVATE_FUTEX, %ecx
+#endif
+ addl $FUTEX_WAKE, %ecx
movl $1, %edx
ENTER_KERNEL
subl $cond_nwaiters, %ebx
@@ -285,11 +300,21 @@ __pthread_cond_timedwait:
1:
.LSbl1:
#if cond_lock == 0
- movl %ebx, %ecx
+ movl %ebx, %edx
#else
- leal cond_lock(%ebx), %ecx
+ leal cond_lock(%ebx), %edx
#endif
- call __lll_mutex_lock_wait
+#if (LLL_SHARED-LLL_PRIVATE) > 255
+ xorl %ecx, %ecx
+#endif
+ cmpl $-1, dep_mutex(%ebx)
+ setne %cl
+ subl $1, %ecx
+ andl $(LLL_SHARED-LLL_PRIVATE), %ecx
+#if LLL_PRIVATE != 0
+ addl $LLL_PRIVATE, %ecx
+#endif
+ call __lll_lock_wait
jmp 2b
/* Unlock in loop requires wakeup. */
@@ -300,17 +325,37 @@ __pthread_cond_timedwait:
#else
leal cond_lock(%ebx), %eax
#endif
- call __lll_mutex_unlock_wake
+#if (LLL_SHARED-LLL_PRIVATE) > 255
+ xorl %ecx, %ecx
+#endif
+ cmpl $-1, dep_mutex(%ebx)
+ setne %cl
+ subl $1, %ecx
+ andl $(LLL_SHARED-LLL_PRIVATE), %ecx
+#if LLL_PRIVATE != 0
+ addl $LLL_PRIVATE, %ecx
+#endif
+ call __lll_unlock_wake
jmp 4b
/* Locking in loop failed. */
5:
#if cond_lock == 0
- movl %ebx, %ecx
+ movl %ebx, %edx
#else
- leal cond_lock(%ebx), %ecx
+ leal cond_lock(%ebx), %edx
#endif
- call __lll_mutex_lock_wait
+#if (LLL_SHARED-LLL_PRIVATE) > 255
+ xorl %ecx, %ecx
+#endif
+ cmpl $-1, dep_mutex(%ebx)
+ setne %cl
+ subl $1, %ecx
+ andl $(LLL_SHARED-LLL_PRIVATE), %ecx
+#if LLL_PRIVATE != 0
+ addl $LLL_PRIVATE, %ecx
+#endif
+ call __lll_lock_wait
jmp 6b
/* Unlock after loop requires wakeup. */
@@ -320,7 +365,17 @@ __pthread_cond_timedwait:
#else
leal cond_lock(%ebx), %eax
#endif
- call __lll_mutex_unlock_wake
+#if (LLL_SHARED-LLL_PRIVATE) > 255
+ xorl %ecx, %ecx
+#endif
+ cmpl $-1, dep_mutex(%ebx)
+ setne %cl
+ subl $1, %ecx
+ andl $(LLL_SHARED-LLL_PRIVATE), %ecx
+#if LLL_PRIVATE != 0
+ addl $LLL_PRIVATE, %ecx
+#endif
+ call __lll_unlock_wake
jmp 11b
/* The initial unlocking of the mutex failed. */
@@ -340,7 +395,17 @@ __pthread_cond_timedwait:
#else
leal cond_lock(%ebx), %eax
#endif
- call __lll_mutex_unlock_wake
+#if (LLL_SHARED-LLL_PRIVATE) > 255
+ xorl %ecx, %ecx
+#endif
+ cmpl $-1, dep_mutex(%ebx)
+ setne %cl
+ subl $1, %ecx
+ andl $(LLL_SHARED-LLL_PRIVATE), %ecx
+#if LLL_PRIVATE != 0
+ addl $LLL_PRIVATE, %ecx
+#endif
+ call __lll_unlock_wake
movl %esi, %eax
jmp 18b
@@ -350,7 +415,7 @@ __pthread_cond_timedwait:
.LSbl4:
19: leal 4(%esp), %ebx
xorl %ecx, %ecx
- movl $SYS_gettimeofday, %eax
+ movl $__NR_gettimeofday, %eax
ENTER_KERNEL
movl %edx, %ebx
@@ -396,11 +461,21 @@ __condvar_tw_cleanup:
jz 1f
#if cond_lock == 0
- movl %ebx, %ecx
+ movl %ebx, %edx
#else
- leal cond_lock(%ebx), %ecx
+ leal cond_lock(%ebx), %edx
+#endif
+#if (LLL_SHARED-LLL_PRIVATE) > 255
+ xorl %ecx, %ecx
#endif
- call __lll_mutex_lock_wait
+ cmpl $-1, dep_mutex(%ebx)
+ setne %cl
+ subl $1, %ecx
+ andl $(LLL_SHARED-LLL_PRIVATE), %ecx
+#if LLL_PRIVATE != 0
+ addl $LLL_PRIVATE, %ecx
+#endif
+ call __lll_lock_wait
1: movl broadcast_seq(%ebx), %eax
cmpl 20(%esp), %eax
@@ -424,7 +499,7 @@ __condvar_tw_cleanup:
7: addl $1, woken_seq(%ebx)
adcl $0, woken_seq+4(%ebx)
-3: subl $(1 << clock_bits), cond_nwaiters(%ebx)
+3: subl $(1 << nwaiters_shift), cond_nwaiters(%ebx)
/* Wake up a thread which wants to destroy the condvar object. */
xorl %edi, %edi
@@ -433,12 +508,23 @@ __condvar_tw_cleanup:
cmpl $0xffffffff, %eax
jne 4f
movl cond_nwaiters(%ebx), %eax
- andl $~((1 << clock_bits) - 1), %eax
+ andl $~((1 << nwaiters_shift) - 1), %eax
jne 4f
addl $cond_nwaiters, %ebx
movl $SYS_futex, %eax
- movl $FUTEX_WAKE, %ecx
+#if FUTEX_PRIVATE_FLAG > 255
+ xorl %ecx, %ecx
+#endif
+ cmpl $-1, dep_mutex-cond_nwaiters(%ebx)
+ sete %cl
+ subl $1, %ecx
+#ifdef __ASSUME_PRIVATE_FUTEX
+ andl $FUTEX_PRIVATE_FLAG, %ecx
+#else
+ andl %gs:PRIVATE_FUTEX, %ecx
+#endif
+ addl $FUTEX_WAKE, %ecx
movl $1, %edx
ENTER_KERNEL
subl $cond_nwaiters, %ebx
@@ -457,13 +543,34 @@ __condvar_tw_cleanup:
#else
leal cond_lock(%ebx), %eax
#endif
- call __lll_mutex_unlock_wake
+#if (LLL_SHARED-LLL_PRIVATE) > 255
+ xorl %ecx, %ecx
+#endif
+ cmpl $-1, dep_mutex(%ebx)
+ setne %cl
+ subl $1, %ecx
+ andl $(LLL_SHARED-LLL_PRIVATE), %ecx
+#if LLL_PRIVATE != 0
+ addl $LLL_PRIVATE, %ecx
+#endif
+ call __lll_unlock_wake
/* Wake up all waiters to make sure no signal gets lost. */
2: testl %edi, %edi
jnz 5f
addl $cond_futex, %ebx
- movl $FUTEX_WAKE, %ecx
+#if FUTEX_PRIVATE_FLAG > 255
+ xorl %ecx, %ecx
+#endif
+ cmpl $-1, dep_mutex-cond_futex(%ebx)
+ sete %cl
+ subl $1, %ecx
+#ifdef __ASSUME_PRIVATE_FUTEX
+ andl $FUTEX_PRIVATE_FLAG, %ecx
+#else
+ andl %gs:PRIVATE_FUTEX, %ecx
+#endif
+ addl $FUTEX_WAKE, %ecx
movl $SYS_futex, %eax
movl $0x7fffffff, %edx
ENTER_KERNEL
@@ -583,12 +690,12 @@ __condvar_tw_cleanup:
.uleb128 20
.byte 0x83 # DW_CFA_offset %ebx
.uleb128 5
- .byte 2 # DW_CFA_advance_loc1
- .byte .Lsubl-.Lpush_ebx
+ .byte 4 # DW_CFA_advance_loc4
+ .4byte .Lsubl-.Lpush_ebx
.byte 14 # DW_CFA_def_cfa_offset
.uleb128 20+FRAME_SIZE
- .byte 3 # DW_CFA_advance_loc2
- .2byte .Laddl-.Lsubl
+ .byte 4 # DW_CFA_advance_loc4
+ .4byte .Laddl-.Lsubl
.byte 14 # DW_CFA_def_cfa_offset
.uleb128 20
.byte 0x40+.Lpop_ebx-.Laddl # DW_CFA_advance_loc+N
@@ -610,7 +717,8 @@ __condvar_tw_cleanup:
.byte 0x40+.LSbl1-.Lpop_edi # DW_CFA_advance_loc+N
.byte 14 # DW_CFA_def_cfa_offset
.uleb128 20
- .byte 0x40+.LSbl2-.LSbl1 # DW_CFA_advance_loc+N
+ .byte 4 # DW_CFA_advance_loc4
+ .4byte .LSbl2-.LSbl1
.byte 14 # DW_CFA_def_cfa_offset
.uleb128 20+FRAME_SIZE
.byte 0x85 # DW_CFA_offset %ebp
@@ -621,14 +729,15 @@ __condvar_tw_cleanup:
.uleb128 4
.byte 0x83 # DW_CFA_offset %ebx
.uleb128 5
- .byte 0x40+.LSbl3-.LSbl2 # DW_CFA_advance_loc+N
+ .byte 4 # DW_CFA_advance_loc4
+ .4byte .LSbl3-.LSbl2
.byte 14 # DW_CFA_def_cfa_offset
.uleb128 20
+ .byte 4 # DW_CFA_advance_loc4
#if defined __NR_clock_gettime && !defined __ASSUME_POSIX_TIMERS
- .byte 0x40+.LSbl4-.LSbl3 # DW_CFA_advance_loc+N
+ .4byte .LSbl4-.LSbl3
#else
- .byte 4 # DW_CFA_advance_loc4
- .long .LSbl5-.LSbl3
+ .4byte .LSbl5-.LSbl3
#endif
.byte 14 # DW_CFA_def_cfa_offset
.uleb128 20+FRAME_SIZE
--- libc/nptl/sysdeps/unix/sysv/linux/i386/i486/pthread_cond_wait.S 9 Sep 2006 11:21:23 -0000 1.34
+++ libc/nptl/sysdeps/unix/sysv/linux/i386/i486/pthread_cond_wait.S 14 Aug 2007 02:24:31 -0000 1.37
@@ -19,18 +19,10 @@
#include <sysdep.h>
#include <shlib-compat.h>
+#include <lowlevellock.h>
#include <lowlevelcond.h>
#include <tcb-offsets.h>
-
-#ifdef UP
-# define LOCK
-#else
-# define LOCK lock
-#endif
-
-#define SYS_futex 240
-#define FUTEX_WAIT 0
-#define FUTEX_WAKE 1
+#include <kernel-features.h>
.text
@@ -80,7 +72,7 @@ __pthread_cond_wait:
addl $1, total_seq(%ebx)
adcl $0, total_seq+4(%ebx)
addl $1, cond_futex(%ebx)
- addl $(1 << clock_bits), cond_nwaiters(%ebx)
+ addl $(1 << nwaiters_shift), cond_nwaiters(%ebx)
#define FRAME_SIZE 16
subl $FRAME_SIZE, %esp
@@ -109,7 +101,20 @@ __pthread_cond_wait:
4: call __pthread_enable_asynccancel
movl %eax, (%esp)
- movl %esi, %ecx /* movl $FUTEX_WAIT, %ecx */
+#if FUTEX_PRIVATE_FLAG > 255
+ xorl %ecx, %ecx
+#endif
+ cmpl $-1, dep_mutex(%ebx)
+ sete %cl
+ subl $1, %ecx
+#ifdef __ASSUME_PRIVATE_FUTEX
+ andl $FUTEX_PRIVATE_FLAG, %ecx
+#else
+ andl %gs:PRIVATE_FUTEX, %ecx
+#endif
+#if FUTEX_WAIT != 0
+ addl $FUTEX_WAIT, %ecx
+#endif
movl %edi, %edx
addl $cond_futex, %ebx
.Ladd_cond_futex:
@@ -157,7 +162,7 @@ __pthread_cond_wait:
adcl $0, woken_seq+4(%ebx)
/* Unlock */
-16: subl $(1 << clock_bits), cond_nwaiters(%ebx)
+16: subl $(1 << nwaiters_shift), cond_nwaiters(%ebx)
/* Wake up a thread which wants to destroy the condvar object. */
movl total_seq(%ebx), %eax
@@ -165,12 +170,23 @@ __pthread_cond_wait:
cmpl $0xffffffff, %eax
jne 17f
movl cond_nwaiters(%ebx), %eax
- andl $~((1 << clock_bits) - 1), %eax
+ andl $~((1 << nwaiters_shift) - 1), %eax
jne 17f
addl $cond_nwaiters, %ebx
movl $SYS_futex, %eax
- movl $FUTEX_WAKE, %ecx
+#if FUTEX_PRIVATE_FLAG > 255
+ xorl %ecx, %ecx
+#endif
+ cmpl $-1, dep_mutex-cond_nwaiters(%ebx)
+ sete %cl
+ subl $1, %ecx
+#ifdef __ASSUME_PRIVATE_FUTEX
+ andl $FUTEX_PRIVATE_FLAG, %ecx
+#else
+ andl %gs:PRIVATE_FUTEX, %ecx
+#endif
+ addl $FUTEX_WAKE, %ecx
movl $1, %edx
ENTER_KERNEL
subl $cond_nwaiters, %ebx
@@ -202,11 +218,21 @@ __pthread_cond_wait:
1:
.LSbl1:
#if cond_lock == 0
- movl %ebx, %ecx
+ movl %ebx, %edx
#else
- leal cond_lock(%ebx), %ecx
+ leal cond_lock(%ebx), %edx
+#endif
+#if (LLL_SHARED-LLL_PRIVATE) > 255
+ xorl %ecx, %ecx
#endif
- call __lll_mutex_lock_wait
+ cmpl $-1, dep_mutex(%ebx)
+ setne %cl
+ subl $1, %ecx
+ andl $(LLL_SHARED-LLL_PRIVATE), %ecx
+#if LLL_PRIVATE != 0
+ addl $LLL_PRIVATE, %ecx
+#endif
+ call __lll_lock_wait
jmp 2b
/* Unlock in loop requires waekup. */
@@ -217,17 +243,37 @@ __pthread_cond_wait:
#else
leal cond_lock(%ebx), %eax
#endif
- call __lll_mutex_unlock_wake
+#if (LLL_SHARED-LLL_PRIVATE) > 255
+ xorl %ecx, %ecx
+#endif
+ cmpl $-1, dep_mutex(%ebx)
+ setne %cl
+ subl $1, %ecx
+ andl $(LLL_SHARED-LLL_PRIVATE), %ecx
+#if LLL_PRIVATE != 0
+ addl $LLL_PRIVATE, %ecx
+#endif
+ call __lll_unlock_wake
jmp 4b
/* Locking in loop failed. */
5:
#if cond_lock == 0
- movl %ebx, %ecx
+ movl %ebx, %edx
#else
- leal cond_lock(%ebx), %ecx
+ leal cond_lock(%ebx), %edx
+#endif
+#if (LLL_SHARED-LLL_PRIVATE) > 255
+ xorl %ecx, %ecx
#endif
- call __lll_mutex_lock_wait
+ cmpl $-1, dep_mutex(%ebx)
+ setne %cl
+ subl $1, %ecx
+ andl $(LLL_SHARED-LLL_PRIVATE), %ecx
+#if LLL_PRIVATE != 0
+ addl $LLL_PRIVATE, %ecx
+#endif
+ call __lll_lock_wait
jmp 6b
/* Unlock after loop requires wakeup. */
@@ -237,7 +283,17 @@ __pthread_cond_wait:
#else
leal cond_lock(%ebx), %eax
#endif
- call __lll_mutex_unlock_wake
+#if (LLL_SHARED-LLL_PRIVATE) > 255
+ xorl %ecx, %ecx
+#endif
+ cmpl $-1, dep_mutex(%ebx)
+ setne %cl
+ subl $1, %ecx
+ andl $(LLL_SHARED-LLL_PRIVATE), %ecx
+#if LLL_PRIVATE != 0
+ addl $LLL_PRIVATE, %ecx
+#endif
+ call __lll_unlock_wake
jmp 11b
/* The initial unlocking of the mutex failed. */
@@ -257,7 +313,17 @@ __pthread_cond_wait:
#else
leal cond_lock(%ebx), %eax
#endif
- call __lll_mutex_unlock_wake
+#if (LLL_SHARED-LLL_PRIVATE) > 255
+ xorl %ecx, %ecx
+#endif
+ cmpl $-1, dep_mutex(%ebx)
+ setne %cl
+ subl $1, %ecx
+ andl $(LLL_SHARED-LLL_PRIVATE), %ecx
+#if LLL_PRIVATE != 0
+ addl $LLL_PRIVATE, %ecx
+#endif
+ call __lll_unlock_wake
movl %esi, %eax
jmp 14b
@@ -287,11 +353,21 @@ __condvar_w_cleanup:
jz 1f
#if cond_lock == 0
- movl %ebx, %ecx
+ movl %ebx, %edx
#else
- leal cond_lock(%ebx), %ecx
+ leal cond_lock(%ebx), %edx
+#endif
+#if (LLL_SHARED-LLL_PRIVATE) > 255
+ xorl %ecx, %ecx
#endif
- call __lll_mutex_lock_wait
+ cmpl $-1, dep_mutex(%ebx)
+ setne %cl
+ subl $1, %ecx
+ andl $(LLL_SHARED-LLL_PRIVATE), %ecx
+#if LLL_PRIVATE != 0
+ addl $LLL_PRIVATE, %ecx
+#endif
+ call __lll_lock_wait
1: movl broadcast_seq(%ebx), %eax
cmpl 12(%esp), %eax
@@ -315,7 +391,7 @@ __condvar_w_cleanup:
7: addl $1, woken_seq(%ebx)
adcl $0, woken_seq+4(%ebx)
-3: subl $(1 << clock_bits), cond_nwaiters(%ebx)
+3: subl $(1 << nwaiters_shift), cond_nwaiters(%ebx)
/* Wake up a thread which wants to destroy the condvar object. */
xorl %edi, %edi
@@ -324,12 +400,23 @@ __condvar_w_cleanup:
cmpl $0xffffffff, %eax
jne 4f
movl cond_nwaiters(%ebx), %eax
- andl $~((1 << clock_bits) - 1), %eax
+ andl $~((1 << nwaiters_shift) - 1), %eax
jne 4f
addl $cond_nwaiters, %ebx
movl $SYS_futex, %eax
- movl $FUTEX_WAKE, %ecx
+#if FUTEX_PRIVATE_FLAG > 255
+ xorl %ecx, %ecx
+#endif
+ cmpl $-1, dep_mutex-cond_nwaiters(%ebx)
+ sete %cl
+ subl $1, %ecx
+#ifdef __ASSUME_PRIVATE_FUTEX
+ andl $FUTEX_PRIVATE_FLAG, %ecx
+#else
+ andl %gs:PRIVATE_FUTEX, %ecx
+#endif
+ addl $FUTEX_WAKE, %ecx
movl $1, %edx
ENTER_KERNEL
subl $cond_nwaiters, %ebx
@@ -348,13 +435,34 @@ __condvar_w_cleanup:
#else
leal cond_lock(%ebx), %eax
#endif
- call __lll_mutex_unlock_wake
+#if (LLL_SHARED-LLL_PRIVATE) > 255
+ xorl %ecx, %ecx
+#endif
+ cmpl $-1, dep_mutex(%ebx)
+ setne %cl
+ subl $1, %ecx
+ andl $(LLL_SHARED-LLL_PRIVATE), %ecx
+#if LLL_PRIVATE != 0
+ addl $LLL_PRIVATE, %ecx
+#endif
+ call __lll_unlock_wake
/* Wake up all waiters to make sure no signal gets lost. */
2: testl %edi, %edi
jnz 5f
addl $cond_futex, %ebx
- movl $FUTEX_WAKE, %ecx
+#if FUTEX_PRIVATE_FLAG > 255
+ xorl %ecx, %ecx
+#endif
+ cmpl $-1, dep_mutex-cond_futex(%ebx)
+ sete %cl
+ subl $1, %ecx
+#ifdef __ASSUME_PRIVATE_FUTEX
+ andl $FUTEX_PRIVATE_FLAG, %ecx
+#else
+ andl %gs:PRIVATE_FUTEX, %ecx
+#endif
+ addl $FUTEX_WAKE, %ecx
movl $SYS_futex, %eax
movl $0x7fffffff, %edx
ENTER_KERNEL
@@ -468,12 +576,12 @@ __condvar_w_cleanup:
.uleb128 16
.byte 0x83 # DW_CFA_offset %ebx
.uleb128 4
- .byte 2 # DW_CFA_advance_loc1
- .byte .Lsubl-.Lpush_ebx
+ .byte 4 # DW_CFA_advance_loc4
+ .4byte .Lsubl-.Lpush_ebx
.byte 14 # DW_CFA_def_cfa_offset
.uleb128 16+FRAME_SIZE
- .byte 2 # DW_CFA_advance_loc1
- .byte .Laddl-.Lsubl
+ .byte 4 # DW_CFA_advance_loc4
+ .4byte .Laddl-.Lsubl
.byte 14 # DW_CFA_def_cfa_offset
.uleb128 16
.byte 0x40+ .Lpop_ebx-.Laddl # DW_CFA_advance_loc+N
@@ -497,13 +605,16 @@ __condvar_w_cleanup:
.uleb128 3
.byte 0x83 # DW_CFA_offset %ebx
.uleb128 4
- .byte 0x40+.LSbl2-.LSbl1 # DW_CFA_advance_loc+N
+ .byte 4 # DW_CFA_advance_loc4
+ .4byte .LSbl2-.LSbl1
.byte 14 # DW_CFA_def_cfa_offset
.uleb128 16+FRAME_SIZE
- .byte 0x40+.LSbl3-.LSbl2 # DW_CFA_advance_loc+N
+ .byte 4 # DW_CFA_advance_loc4
+ .4byte .LSbl3-.LSbl2
.byte 14 # DW_CFA_def_cfa_offset
.uleb128 16
- .byte 0x40+.LSbl4-.LSbl3 # DW_CFA_advance_loc+N
+ .byte 4 # DW_CFA_advance_loc4
+ .4byte .LSbl4-.LSbl3
.byte 14 # DW_CFA_def_cfa_offset
.uleb128 16+FRAME_SIZE
.align 4
--- libc/nptl/sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_rdlock.S 21 Sep 2003 17:40:25 -0000 1.10
+++ libc/nptl/sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_rdlock.S 14 Aug 2007 02:24:49 -0000 1.16
@@ -18,19 +18,10 @@
02111-1307 USA. */
#include <sysdep.h>
+#include <lowlevellock.h>
#include <lowlevelrwlock.h>
#include <pthread-errnos.h>
-
-
-#define SYS_futex 240
-#define FUTEX_WAIT 0
-#define FUTEX_WAKE 1
-
-#ifndef UP
-# define LOCK lock
-#else
-# define LOCK
-#endif
+#include <kernel-features.h>
.text
@@ -61,7 +52,7 @@ __pthread_rwlock_rdlock:
jne 14f
cmpl $0, WRITERS_QUEUED(%ebx)
je 5f
- cmpl $0, FLAGS(%ebx)
+ cmpb $0, FLAGS(%ebx)
je 5f
3: addl $1, READERS_QUEUED(%ebx)
@@ -77,8 +68,18 @@ __pthread_rwlock_rdlock:
#endif
jne 10f
-11: addl $READERS_WAKEUP, %ebx
- movl %esi, %ecx /* movl $FUTEX_WAIT, %ecx */
+11:
+#ifdef __ASSUME_PRIVATE_FUTEX
+ movzbl PSHARED(%ebx), %ecx
+ xorl $FUTEX_PRIVATE_FLAG|FUTEX_WAIT, %ecx
+#else
+ movzbl PSHARED(%ebx), %ecx
+# if FUTEX_WAIT != 0
+ orl $FUTEX_WAIT, %ecx
+# endif
+ xorl %gs:PRIVATE_FUTEX, %ecx
+#endif
+ addl $READERS_WAKEUP, %ebx
movl $SYS_futex, %eax
ENTER_KERNEL
@@ -98,7 +99,7 @@ __pthread_rwlock_rdlock:
13: subl $1, READERS_QUEUED(%ebx)
jmp 2b
-5: xorl %ecx, %ecx
+5: xorl %edx, %edx
addl $1, NR_READERS(%ebx)
je 8f
9: LOCK
@@ -110,24 +111,25 @@ __pthread_rwlock_rdlock:
jne 6f
7:
- movl %ecx, %eax
+ movl %edx, %eax
popl %ebx
popl %esi
ret
1:
#if MUTEX == 0
- movl %ebx, %ecx
+ movl %ebx, %edx
#else
- leal MUTEX(%ebx), %ecx
+ leal MUTEX(%ebx), %edx
#endif
- call __lll_mutex_lock_wait
+ movzbl PSHARED(%ebx), %ecx
+ call __lll_lock_wait
jmp 2b
14: cmpl %gs:TID, %eax
jne 3b
/* Deadlock detected. */
- movl $EDEADLK, %ecx
+ movl $EDEADLK, %edx
jmp 9b
6:
@@ -136,17 +138,18 @@ __pthread_rwlock_rdlock:
#else
leal MUTEX(%ebx), %eax
#endif
- call __lll_mutex_unlock_wake
+ movzbl PSHARED(%ebx), %ecx
+ call __lll_unlock_wake
jmp 7b
/* Overflow. */
8: subl $1, NR_READERS(%ebx)
- movl $EAGAIN, %ecx
+ movl $EAGAIN, %edx
jmp 9b
/* Overflow. */
4: subl $1, READERS_QUEUED(%ebx)
- movl $EAGAIN, %ecx
+ movl $EAGAIN, %edx
jmp 9b
10:
@@ -155,16 +158,18 @@ __pthread_rwlock_rdlock:
#else
leal MUTEX(%ebx), %eax
#endif
- call __lll_mutex_unlock_wake
+ movzbl PSHARED(%ebx), %ecx
+ call __lll_unlock_wake
jmp 11b
12:
#if MUTEX == 0
- movl %ebx, %ecx
+ movl %ebx, %edx
#else
- leal MUTEX(%ebx), %ecx
+ leal MUTEX(%ebx), %edx
#endif
- call __lll_mutex_lock_wait
+ movzbl PSHARED(%ebx), %ecx
+ call __lll_lock_wait
jmp 13b
.size __pthread_rwlock_rdlock,.-__pthread_rwlock_rdlock
--- libc/nptl/sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_timedrdlock.S 21 Sep 2003 17:40:25 -0000 1.10
+++ libc/nptl/sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_timedrdlock.S 14 Aug 2007 02:25:44 -0000 1.17
@@ -18,20 +18,10 @@
02111-1307 USA. */
#include <sysdep.h>
+#include <lowlevellock.h>
#include <lowlevelrwlock.h>
#include <pthread-errnos.h>
-
-
-#define SYS_gettimeofday __NR_gettimeofday
-#define SYS_futex 240
-#define FUTEX_WAIT 0
-#define FUTEX_WAKE 1
-
-#ifndef UP
-# define LOCK lock
-#else
-# define LOCK
-#endif
+#include <kernel-features.h>
.text
@@ -65,7 +55,7 @@ pthread_rwlock_timedrdlock:
jne 14f
cmpl $0, WRITERS_QUEUED(%ebp)
je 5f
- cmpl $0, FLAGS(%ebp)
+ cmpb $0, FLAGS(%ebp)
je 5f
/* Check the value of the timeout parameter. */
@@ -88,7 +78,7 @@ pthread_rwlock_timedrdlock:
/* Get current time. */
11: movl %esp, %ebx
xorl %ecx, %ecx
- movl $SYS_gettimeofday, %eax
+ movl $__NR_gettimeofday, %eax
ENTER_KERNEL
/* Compute relative timeout. */
@@ -108,8 +98,18 @@ pthread_rwlock_timedrdlock:
/* Futex call. */
movl %ecx, (%esp) /* Store relative timeout. */
movl %edx, 4(%esp)
+
movl %esi, %edx
- xorl %ecx, %ecx /* movl $FUTEX_WAIT, %ecx */
+#ifdef __ASSUME_PRIVATE_FUTEX
+ movzbl PSHARED(%ebp), %ecx
+ xorl $FUTEX_PRIVATE_FLAG|FUTEX_WAIT, %ecx
+#else
+ movzbl PSHARED(%ebp), %ecx
+# if FUTEX_WAIT != 0
+ orl $FUTEX_WAIT, %ecx
+# endif
+ xorl %gs:PRIVATE_FUTEX, %ecx
+#endif
movl %esp, %esi
leal READERS_WAKEUP(%ebp), %ebx
movl $SYS_futex, %eax
@@ -132,11 +132,11 @@ pthread_rwlock_timedrdlock:
cmpl $-ETIMEDOUT, %esi
jne 2b
-18: movl $ETIMEDOUT, %ecx
+18: movl $ETIMEDOUT, %edx
jmp 9f
-5: xorl %ecx, %ecx
+5: xorl %edx, %edx
addl $1, NR_READERS(%ebp)
je 8f
9: LOCK
@@ -147,7 +147,7 @@ pthread_rwlock_timedrdlock:
#endif
jne 6f
-7: movl %ecx, %eax
+7: movl %edx, %eax
addl $8, %esp
popl %ebp
@@ -158,16 +158,17 @@ pthread_rwlock_timedrdlock:
1:
#if MUTEX == 0
- movl %ebp, %ecx
+ movl %ebp, %edx
#else
- leal MUTEX(%ebp), %ecx
+ leal MUTEX(%ebp), %edx
#endif
- call __lll_mutex_lock_wait
+ movzbl PSHARED(%ebp), %ecx
+ call __lll_lock_wait
jmp 2b
14: cmpl %gs:TID, %eax
jne 3b
- movl $EDEADLK, %ecx
+ movl $EDEADLK, %edx
jmp 9b
6:
@@ -176,17 +177,18 @@ pthread_rwlock_timedrdlock:
#else
leal MUTEX(%ebp), %eax
#endif
- call __lll_mutex_unlock_wake
+ movzbl PSHARED(%ebp), %ecx
+ call __lll_unlock_wake
jmp 7b
/* Overflow. */
8: subl $1, NR_READERS(%ebp)
- movl $EAGAIN, %ecx
+ movl $EAGAIN, %edx
jmp 9b
/* Overflow. */
4: subl $1, READERS_QUEUED(%ebp)
- movl $EAGAIN, %ecx
+ movl $EAGAIN, %edx
jmp 9b
10:
@@ -195,21 +197,23 @@ pthread_rwlock_timedrdlock:
#else
leal MUTEX(%ebp), %eax
#endif
- call __lll_mutex_unlock_wake
+ movzbl PSHARED(%ebp), %ecx
+ call __lll_unlock_wake
jmp 11b
12:
#if MUTEX == 0
- movl %ebp, %ecx
+ movl %ebp, %edx
#else
- leal MUTEX(%ebp), %ecx
+ leal MUTEX(%ebp), %edx
#endif
- call __lll_mutex_lock_wait
+ movzbl PSHARED(%ebp), %ecx
+ call __lll_lock_wait
jmp 13b
16: movl $-ETIMEDOUT, %esi
jmp 17b
-19: movl $EINVAL, %ecx
+19: movl $EINVAL, %edx
jmp 9b
.size pthread_rwlock_timedrdlock,.-pthread_rwlock_timedrdlock
--- libc/nptl/sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_timedwrlock.S 21 Sep 2003 17:40:25 -0000 1.11
+++ libc/nptl/sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_timedwrlock.S 14 Aug 2007 02:24:58 -0000 1.18
@@ -18,20 +18,10 @@
02111-1307 USA. */
#include <sysdep.h>
+#include <lowlevellock.h>
#include <lowlevelrwlock.h>
#include <pthread-errnos.h>
-
-
-#define SYS_gettimeofday __NR_gettimeofday
-#define SYS_futex 240
-#define FUTEX_WAIT 0
-#define FUTEX_WAKE 1
-
-#ifndef UP
-# define LOCK lock
-#else
-# define LOCK
-#endif
+#include <kernel-features.h>
.text
@@ -86,7 +76,7 @@ pthread_rwlock_timedwrlock:
/* Get current time. */
11: movl %esp, %ebx
xorl %ecx, %ecx
- movl $SYS_gettimeofday, %eax
+ movl $__NR_gettimeofday, %eax
ENTER_KERNEL
/* Compute relative timeout. */
@@ -106,8 +96,18 @@ pthread_rwlock_timedwrlock:
/* Futex call. */
movl %ecx, (%esp) /* Store relative timeout. */
movl %edx, 4(%esp)
+
movl %esi, %edx
- xorl %ecx, %ecx /* movl $FUTEX_WAIT, %ecx */
+#ifdef __ASSUME_PRIVATE_FUTEX
+ movzbl PSHARED(%ebp), %ecx
+ xorl $FUTEX_PRIVATE_FLAG|FUTEX_WAIT, %ecx
+#else
+ movzbl PSHARED(%ebp), %ecx
+# if FUTEX_WAIT != 0
+ orl $FUTEX_WAIT, %ecx
+# endif
+ xorl %gs:PRIVATE_FUTEX, %ecx
+#endif
movl %esp, %esi
leal WRITERS_WAKEUP(%ebp), %ebx
movl $SYS_futex, %eax
@@ -130,11 +130,11 @@ pthread_rwlock_timedwrlock:
cmpl $-ETIMEDOUT, %esi
jne 2b
-18: movl $ETIMEDOUT, %ecx
+18: movl $ETIMEDOUT, %edx
jmp 9f
-5: xorl %ecx, %ecx
+5: xorl %edx, %edx
movl %gs:TID, %eax
movl %eax, WRITER(%ebp)
9: LOCK
@@ -145,7 +145,7 @@ pthread_rwlock_timedwrlock:
#endif
jne 6f
-7: movl %ecx, %eax
+7: movl %edx, %eax
addl $8, %esp
popl %ebp
@@ -156,16 +156,17 @@ pthread_rwlock_timedwrlock:
1:
#if MUTEX == 0
- movl %ebp, %ecx
+ movl %ebp, %edx
#else
- leal MUTEX(%ebp), %ecx
+ leal MUTEX(%ebp), %edx
#endif
- call __lll_mutex_lock_wait
+ movzbl PSHARED(%ebp), %ecx
+ call __lll_lock_wait
jmp 2b
14: cmpl %gs:TID, %eax
jne 3b
-20: movl $EDEADLK, %ecx
+20: movl $EDEADLK, %edx
jmp 9b
6:
@@ -174,12 +175,13 @@ pthread_rwlock_timedwrlock:
#else
leal MUTEX(%ebp), %eax
#endif
- call __lll_mutex_unlock_wake
+ movzbl PSHARED(%ebp), %ecx
+ call __lll_unlock_wake
jmp 7b
/* Overflow. */
4: subl $1, WRITERS_QUEUED(%ebp)
- movl $EAGAIN, %ecx
+ movl $EAGAIN, %edx
jmp 9b
10:
@@ -188,21 +190,23 @@ pthread_rwlock_timedwrlock:
#else
leal MUTEX(%ebp), %eax
#endif
- call __lll_mutex_unlock_wake
+ movzbl PSHARED(%ebp), %ecx
+ call __lll_unlock_wake
jmp 11b
12:
#if MUTEX == 0
- movl %ebp, %ecx
+ movl %ebp, %edx
#else
- leal MUTEX(%ebp), %ecx
+ leal MUTEX(%ebp), %edx
#endif
- call __lll_mutex_lock_wait
+ movzbl PSHARED(%ebp), %ecx
+ call __lll_lock_wait
jmp 13b
16: movl $-ETIMEDOUT, %esi
jmp 17b
-19: movl $EINVAL, %ecx
+19: movl $EINVAL, %edx
jmp 9b
.size pthread_rwlock_timedwrlock,.-pthread_rwlock_timedwrlock
--- libc/nptl/sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_unlock.S 21 Sep 2003 17:40:25 -0000 1.7
+++ libc/nptl/sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_unlock.S 14 Aug 2007 02:25:06 -0000 1.14
@@ -18,18 +18,9 @@
02111-1307 USA. */
#include <sysdep.h>
+#include <lowlevellock.h>
#include <lowlevelrwlock.h>
-
-
-#define SYS_futex 240
-#define FUTEX_WAIT 0
-#define FUTEX_WAKE 1
-
-#ifndef UP
-# define LOCK lock
-#else
-# define LOCK
-#endif
+#include <kernel-features.h>
.text
@@ -61,9 +52,8 @@ __pthread_rwlock_unlock:
5: movl $0, WRITER(%edi)
- movl $1, %ecx
+ movl $1, %edx
leal WRITERS_WAKEUP(%edi), %ebx
- movl %ecx, %edx
cmpl $0, WRITERS_QUEUED(%edi)
jne 0f
@@ -83,7 +73,16 @@ __pthread_rwlock_unlock:
#endif
jne 7f
-8: movl $SYS_futex, %eax
+8:
+#ifdef __ASSUME_PRIVATE_FUTEX
+ movzbl PSHARED(%edi), %ecx
+ xorl $FUTEX_PRIVATE_FLAG|FUTEX_WAKE, %ecx
+#else
+ movzbl PSHARED(%edi), %ecx
+ orl $FUTEX_WAKE, %ecx
+ xorl %gs:PRIVATE_FUTEX, %ecx
+#endif
+ movl $SYS_futex, %eax
ENTER_KERNEL
xorl %eax, %eax
@@ -107,11 +106,12 @@ __pthread_rwlock_unlock:
1:
#if MUTEX == 0
- movl %edi, %ecx
+ movl %edi, %edx
#else
- leal MUTEX(%edi), %ecx
+ leal MUTEX(%edi), %edx
#endif
- call __lll_mutex_lock_wait
+ movzbl PSHARED(%edi), %ecx
+ call __lll_lock_wait
jmp 2b
3:
@@ -120,7 +120,8 @@ __pthread_rwlock_unlock:
#else
leal MUTEX(%edi), %eax
#endif
- call __lll_mutex_unlock_wake
+ movzbl PSHARED(%edi), %ecx
+ call __lll_unlock_wake
jmp 4b
7:
@@ -129,7 +130,8 @@ __pthread_rwlock_unlock:
#else
leal MUTEX(%edi), %eax
#endif
- call __lll_mutex_unlock_wake
+ movzbl PSHARED(%edi), %ecx
+ call __lll_unlock_wake
jmp 8b
.size __pthread_rwlock_unlock,.-__pthread_rwlock_unlock
--- libc/nptl/sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_wrlock.S 21 Sep 2003 17:40:25 -0000 1.10
+++ libc/nptl/sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_wrlock.S 14 Aug 2007 02:25:27 -0000 1.16
@@ -18,19 +18,10 @@
02111-1307 USA. */
#include <sysdep.h>
+#include <lowlevellock.h>
#include <lowlevelrwlock.h>
#include <pthread-errnos.h>
-
-
-#define SYS_futex 240
-#define FUTEX_WAIT 0
-#define FUTEX_WAKE 1
-
-#ifndef UP
-# define LOCK lock
-#else
-# define LOCK
-#endif
+#include <kernel-features.h>
.text
@@ -75,8 +66,18 @@ __pthread_rwlock_wrlock:
#endif
jne 10f
-11: addl $WRITERS_WAKEUP, %ebx
- movl %esi, %ecx /* movl $FUTEX_WAIT, %ecx */
+11:
+#ifdef __ASSUME_PRIVATE_FUTEX
+ movzbl PSHARED(%ebx), %ecx
+ xorl $FUTEX_PRIVATE_FLAG|FUTEX_WAIT, %ecx
+#else
+ movzbl PSHARED(%ebx), %ecx
+# if FUTEX_WAIT != 0
+ orl $FUTEX_WAIT, %ecx
+# endif
+ xorl %gs:PRIVATE_FUTEX, %ecx
+#endif
+ addl $WRITERS_WAKEUP, %ebx
movl $SYS_futex, %eax
ENTER_KERNEL
@@ -96,7 +97,7 @@ __pthread_rwlock_wrlock:
13: subl $1, WRITERS_QUEUED(%ebx)
jmp 2b
-5: xorl %ecx, %ecx
+5: xorl %edx, %edx
movl %gs:TID, %eax
movl %eax, WRITER(%ebx)
9: LOCK
@@ -108,23 +109,24 @@ __pthread_rwlock_wrlock:
jne 6f
7:
- movl %ecx, %eax
+ movl %edx, %eax
popl %ebx
popl %esi
ret
1:
#if MUTEX == 0
- movl %ebx, %ecx
+ movl %ebx, %edx
#else
- leal MUTEX(%ebx), %ecx
+ leal MUTEX(%ebx), %edx
#endif
- call __lll_mutex_lock_wait
+ movzbl PSHARED(%ebx), %ecx
+ call __lll_lock_wait
jmp 2b
14: cmpl %gs:TID , %eax
jne 3b
- movl $EDEADLK, %ecx
+ movl $EDEADLK, %edx
jmp 9b
6:
@@ -133,11 +135,12 @@ __pthread_rwlock_wrlock:
#else
leal MUTEX(%ebx), %eax
#endif
- call __lll_mutex_unlock_wake
+ movzbl PSHARED(%ebx), %ecx
+ call __lll_unlock_wake
jmp 7b
4: subl $1, WRITERS_QUEUED(%ebx)
- movl $EAGAIN, %ecx
+ movl $EAGAIN, %edx
jmp 9b
10:
@@ -146,16 +149,18 @@ __pthread_rwlock_wrlock:
#else
leal MUTEX(%ebx), %eax
#endif
- call __lll_mutex_unlock_wake
+ movzbl PSHARED(%ebx), %ecx
+ call __lll_unlock_wake
jmp 11b
12:
#if MUTEX == 0
- movl %ebx, %ecx
+ movl %ebx, %edx
#else
- leal MUTEX(%ebx), %ecx
+ leal MUTEX(%ebx), %edx
#endif
- call __lll_mutex_lock_wait
+ movzbl PSHARED(%ebx), %ecx
+ call __lll_lock_wait
jmp 13b
.size __pthread_rwlock_wrlock,.-__pthread_rwlock_wrlock
--- libc/nptl/sysdeps/unix/sysv/linux/i386/i486/sem_post.S 15 May 2007 06:31:57 -0000 1.9
+++ libc/nptl/sysdeps/unix/sysv/linux/i386/i486/sem_post.S 17 Oct 2007 14:34:44 -0000 1.13
@@ -20,15 +20,8 @@
#include <sysdep.h>
#include <shlib-compat.h>
#include <pthread-errnos.h>
-
-#ifndef UP
-# define LOCK lock
-#else
-# define
-#endif
-
-#define SYS_futex 240
-#define FUTEX_WAKE 1
+#include <structsem.h>
+#include <lowlevellock.h>
.text
@@ -40,19 +33,27 @@ __new_sem_post:
pushl %ebx
movl 8(%esp), %ebx
- movl $1, %edx
+
LOCK
- xaddl %edx, (%ebx)
+#if VALUE == 0
+ addl $1, (%ebx)
+#else
+ addl $1, VALUE(%ebx)
+#endif
+
+ cmpl $0, NWAITERS(%ebx)
+ je 2f
- movl $SYS_futex, %eax
movl $FUTEX_WAKE, %ecx
- addl $1, %edx
+ orl PRIVATE(%ebx), %ecx
+ movl $1, %edx
+ movl $SYS_futex, %eax
ENTER_KERNEL
testl %eax, %eax
js 1f
- xorl %eax, %eax
+2: xorl %eax, %eax
popl %ebx
ret
@@ -85,7 +86,26 @@ __new_sem_post:
versioned_symbol(libpthread, __new_sem_post, sem_post, GLIBC_2_1)
#if SHLIB_COMPAT(libpthread, GLIBC_2_0, GLIBC_2_1)
.global __old_sem_post
-__old_sem_post = __new_sem_post
+ .type __old_sem_post,@function
+__old_sem_post:
+ pushl %ebx
+
+ movl 8(%esp), %ebx
+ LOCK
+ addl $1, (%ebx)
+
+ movl $SYS_futex, %eax
+ movl $FUTEX_WAKE, %ecx
+ movl $1, %edx
+ ENTER_KERNEL
+
+ testl %eax, %eax
+ js 1b
+
+ xorl %eax, %eax
+ popl %ebx
+ ret
+ .size __old_sem_post,.-__old_sem_post
compat_symbol(libpthread, __old_sem_post, sem_post, GLIBC_2_0)
#endif
--- libc/nptl/sysdeps/unix/sysv/linux/i386/i486/sem_timedwait.S 9 Apr 2006 02:42:29 -0000 1.8
+++ libc/nptl/sysdeps/unix/sysv/linux/i386/i486/sem_timedwait.S 1 Aug 2007 04:29:40 -0000 1.12
@@ -20,16 +20,13 @@
#include <sysdep.h>
#include <shlib-compat.h>
#include <pthread-errnos.h>
+#include <structsem.h>
+#include <lowlevellock.h>
-#ifndef UP
-# define LOCK lock
-#else
-# define
-#endif
-#define SYS_gettimeofday __NR_gettimeofday
-#define SYS_futex 240
-#define FUTEX_WAKE 1
+#if VALUE != 0
+# error "code needs to be rewritten for VALUE != 0"
+#endif
.text
@@ -37,14 +34,8 @@
.globl sem_timedwait
.type sem_timedwait,@function
.align 16
- cfi_startproc
sem_timedwait:
- /* First check for cancellation. */
- movl %gs:CANCELHANDLING, %eax
- andl $0xfffffff9, %eax
- cmpl $8, %eax
- je 10f
-
+.LSTARTCODE:
movl 4(%esp), %ecx
movl (%ecx), %eax
@@ -61,28 +52,28 @@ sem_timedwait:
/* Check whether the timeout value is valid. */
1: pushl %esi
- cfi_adjust_cfa_offset(4)
+.Lpush_esi:
pushl %edi
- cfi_adjust_cfa_offset(4)
+.Lpush_edi:
pushl %ebx
- cfi_adjust_cfa_offset(4)
+.Lpush_ebx:
subl $12, %esp
- cfi_adjust_cfa_offset(12)
+.Lsub_esp:
movl 32(%esp), %edi
- cfi_offset(7, -12) /* %edi */
/* Check for invalid nanosecond field. */
cmpl $1000000000, 4(%edi)
movl $EINVAL, %esi
- cfi_offset(6, -8) /* %esi */
jae 6f
- cfi_offset(3, -16) /* %ebx */
+ LOCK
+ incl NWAITERS(%ecx)
+
7: xorl %ecx, %ecx
movl %esp, %ebx
movl %ecx, %edx
- movl $SYS_gettimeofday, %eax
+ movl $__NR_gettimeofday, %eax
ENTER_KERNEL
/* Compute relative timeout. */
@@ -103,19 +94,26 @@ sem_timedwait:
movl %ecx, (%esp) /* Store relative timeout. */
movl %edx, 4(%esp)
+.LcleanupSTART:
call __pthread_enable_asynccancel
movl %eax, 8(%esp)
- movl 28(%esp), %ebx
- xorl %ecx, %ecx
+ movl 28(%esp), %ebx /* Load semaphore address. */
+#if FUTEX_WAIT == 0
+ movl PRIVATE(%ebx), %ecx
+#else
+ movl $FUTEX_WAIT, %ecx
+ orl PRIVATE(%ebx), %ecx
+#endif
movl %esp, %esi
- movl $SYS_futex, %eax
xorl %edx, %edx
+ movl $SYS_futex, %eax
ENTER_KERNEL
movl %eax, %esi
movl 8(%esp), %eax
call __pthread_disable_asynccancel
+.LcleanupEND:
testl %esi, %esi
je 9f
@@ -131,24 +129,22 @@ sem_timedwait:
cmpxchgl %ecx, (%ebx)
jne 8b
- addl $12, %esp
- cfi_adjust_cfa_offset(-12)
xorl %eax, %eax
+
+10: LOCK
+ decl NWAITERS(%ebx)
+
+ addl $12, %esp
+.Ladd_esp:
popl %ebx
- cfi_adjust_cfa_offset(-4)
- cfi_restore(3)
+.Lpop_ebx:
popl %edi
- cfi_adjust_cfa_offset(-4)
- cfi_restore(7)
+.Lpop_edi:
popl %esi
- cfi_adjust_cfa_offset(-4)
- cfi_restore(6)
+.Lpop_esi:
ret
- cfi_adjust_cfa_offset(24)
- cfi_offset(6, -8) /* %esi */
- cfi_offset(7, -12) /* %edi */
- cfi_offset(3, -16) /* %ebx */
+.Lafter_ret:
3: negl %esi
6:
#ifdef PIC
@@ -172,25 +168,163 @@ sem_timedwait:
movl %esi, (%eax)
#endif
- addl $12, %esp
- cfi_adjust_cfa_offset(-12)
+ movl 28(%esp), %ebx /* Load semaphore address. */
orl $-1, %eax
- popl %ebx
- cfi_adjust_cfa_offset(-4)
- cfi_restore(3)
- popl %edi
- cfi_adjust_cfa_offset(-4)
- cfi_restore(7)
- popl %esi
- cfi_adjust_cfa_offset(-4)
- cfi_restore(6)
- ret
+ jmp 10b
+ .size sem_timedwait,.-sem_timedwait
-10: /* Canceled. */
- movl $0xffffffff, %gs:RESULT
+
+ .type sem_wait_cleanup,@function
+sem_wait_cleanup:
LOCK
- orl $0x10, %gs:CANCELHANDLING
- movl %gs:CLEANUP_JMP_BUF, %eax
- jmp HIDDEN_JUMPTARGET (__pthread_unwind)
- cfi_endproc
- .size sem_timedwait,.-sem_timedwait
+ decl NWAITERS(%ebx)
+ movl %eax, (%esp)
+.LcallUR:
+ call _Unwind_Resume@PLT
+ hlt
+.LENDCODE:
+ .size sem_wait_cleanup,.-sem_wait_cleanup
+
+
+ .section .gcc_except_table,"a",@progbits
+.LexceptSTART:
+ .byte 0xff # @LPStart format (omit)
+ .byte 0xff # @TType format (omit)
+ .byte 0x01 # call-site format
+ # DW_EH_PE_uleb128
+ .uleb128 .Lcstend-.Lcstbegin
+.Lcstbegin:
+ .uleb128 .LcleanupSTART-.LSTARTCODE
+ .uleb128 .LcleanupEND-.LcleanupSTART
+ .uleb128 sem_wait_cleanup-.LSTARTCODE
+ .uleb128 0
+ .uleb128 .LcallUR-.LSTARTCODE
+ .uleb128 .LENDCODE-.LcallUR
+ .uleb128 0
+ .uleb128 0
+.Lcstend:
+
+
+ .section .eh_frame,"a",@progbits
+.LSTARTFRAME:
+ .long .LENDCIE-.LSTARTCIE # Length of the CIE.
+.LSTARTCIE:
+ .long 0 # CIE ID.
+ .byte 1 # Version number.
+#ifdef SHARED
+ .string "zPLR" # NUL-terminated augmentation
+ # string.
+#else
+ .string "zPL" # NUL-terminated augmentation
+ # string.
+#endif
+ .uleb128 1 # Code alignment factor.
+ .sleb128 -4 # Data alignment factor.
+ .byte 8 # Return address register
+ # column.
+#ifdef SHARED
+ .uleb128 7 # Augmentation value length.
+ .byte 0x9b # Personality: DW_EH_PE_pcrel
+ # + DW_EH_PE_sdata4
+ # + DW_EH_PE_indirect
+ .long DW.ref.__gcc_personality_v0-.
+ .byte 0x1b # LSDA Encoding: DW_EH_PE_pcrel
+ # + DW_EH_PE_sdata4.
+ .byte 0x1b # FDE Encoding: DW_EH_PE_pcrel
+ # + DW_EH_PE_sdata4.
+#else
+ .uleb128 6 # Augmentation value length.
+ .byte 0x0 # Personality: absolute
+ .long __gcc_personality_v0
+ .byte 0x0 # LSDA Encoding: absolute
+#endif
+ .byte 0x0c # DW_CFA_def_cfa
+ .uleb128 4
+ .uleb128 4
+ .byte 0x88 # DW_CFA_offset, column 0x10
+ .uleb128 1
+ .align 4
+.LENDCIE:
+
+ .long .LENDFDE-.LSTARTFDE # Length of the FDE.
+.LSTARTFDE:
+ .long .LSTARTFDE-.LSTARTFRAME # CIE pointer.
+#ifdef SHARED
+ .long .LSTARTCODE-. # PC-relative start address
+ # of the code.
+#else
+ .long .LSTARTCODE # Start address of the code.
+#endif
+ .long .LENDCODE-.LSTARTCODE # Length of the code.
+ .uleb128 4 # Augmentation size
+#ifdef SHARED
+ .long .LexceptSTART-.
+#else
+ .long .LexceptSTART
+#endif
+
+ .byte 4 # DW_CFA_advance_loc4
+ .long .Lpush_esi-.LSTARTCODE
+ .byte 14 # DW_CFA_def_cfa_offset
+ .uleb128 8
+ .byte 0x86 # DW_CFA_offset %esi
+ .uleb128 2
+ .byte 4 # DW_CFA_advance_loc4
+ .long .Lpush_edi-.Lpush_esi
+ .byte 14 # DW_CFA_def_cfa_offset
+ .uleb128 12
+ .byte 0x87 # DW_CFA_offset %edi
+ .uleb128 3
+ .byte 4 # DW_CFA_advance_loc4
+ .long .Lpush_ebx-.Lpush_edi
+ .byte 14 # DW_CFA_def_cfa_offset
+ .uleb128 16
+ .byte 0x83 # DW_CFA_offset %ebx
+ .uleb128 4
+ .byte 4 # DW_CFA_advance_loc4
+ .long .Lsub_esp-.Lpush_ebx
+ .byte 14 # DW_CFA_def_cfa_offset
+ .uleb128 28
+ .byte 4 # DW_CFA_advance_loc4
+ .long .Ladd_esp-.Lsub_esp
+ .byte 14 # DW_CFA_def_cfa_offset
+ .uleb128 16
+ .byte 4 # DW_CFA_advance_loc4
+ .long .Lpop_ebx-.Ladd_esp
+ .byte 14 # DW_CFA_def_cfa_offset
+ .uleb128 12
+ .byte 0xc3 # DW_CFA_restore %ebx
+ .byte 4 # DW_CFA_advance_loc4
+ .long .Lpop_edi-.Lpop_ebx
+ .byte 14 # DW_CFA_def_cfa_offset
+ .uleb128 8
+ .byte 0xc7 # DW_CFA_restore %edi
+ .byte 4 # DW_CFA_advance_loc4
+ .long .Lpop_esi-.Lpop_edi
+ .byte 14 # DW_CFA_def_cfa_offset
+ .uleb128 4
+ .byte 0xc6 # DW_CFA_restore %esi
+ .byte 4 # DW_CFA_advance_loc4
+ .long .Lafter_ret-.Lpop_esi
+ .byte 14 # DW_CFA_def_cfa_offset
+ .uleb128 28
+ .byte 0x86 # DW_CFA_offset %esi
+ .uleb128 2
+ .byte 0x87 # DW_CFA_offset %edi
+ .uleb128 3
+ .byte 0x83 # DW_CFA_offset %ebx
+ .uleb128 4
+ .align 4
+.LENDFDE:
+
+
+#ifdef SHARED
+ .hidden DW.ref.__gcc_personality_v0
+ .weak DW.ref.__gcc_personality_v0
+ .section .gnu.linkonce.d.DW.ref.__gcc_personality_v0,"aw",@progbits
+ .align 4
+ .type DW.ref.__gcc_personality_v0, @object
+ .size DW.ref.__gcc_personality_v0, 4
+DW.ref.__gcc_personality_v0:
+ .long __gcc_personality_v0
+#endif
--- libc/nptl/sysdeps/unix/sysv/linux/i386/i486/sem_trywait.S 9 Apr 2006 02:42:29 -0000 1.6
+++ libc/nptl/sysdeps/unix/sysv/linux/i386/i486/sem_trywait.S 1 Aug 2007 04:24:26 -0000 1.7
@@ -20,12 +20,7 @@
#include <sysdep.h>
#include <shlib-compat.h>
#include <pthread-errnos.h>
-
-#ifndef UP
-# define LOCK lock
-#else
-# define
-#endif
+#include <lowlevellock.h>
.text
--- libc/nptl/sysdeps/unix/sysv/linux/i386/i486/sem_wait.S 9 Apr 2006 02:42:29 -0000 1.8
+++ libc/nptl/sysdeps/unix/sysv/linux/i386/i486/sem_wait.S 1 Aug 2007 04:29:58 -0000 1.11
@@ -20,30 +20,262 @@
#include <sysdep.h>
#include <shlib-compat.h>
#include <pthread-errnos.h>
+#include <structsem.h>
+#include <lowlevellock.h>
-#ifndef UP
-# define LOCK lock
-#else
-# define
-#endif
-
-#define SYS_futex 240
-#define FUTEX_WAKE 1
+#if VALUE != 0
+# error "code needs to be rewritten for VALUE != 0"
+#endif
.text
.globl __new_sem_wait
.type __new_sem_wait,@function
.align 16
- cfi_startproc
__new_sem_wait:
- /* First check for cancellation. */
- movl %gs:CANCELHANDLING, %eax
- andl $0xfffffff9, %eax
- cmpl $8, %eax
- je 5f
+.LSTARTCODE:
+ pushl %ebx
+.Lpush_ebx:
+ pushl %esi
+.Lpush_esi:
+ subl $4, %esp
+.Lsub_esp:
+
+ movl 16(%esp), %ebx
+
+ movl (%ebx), %eax
+2: testl %eax, %eax
+ je 1f
+
+ leal -1(%eax), %edx
+ LOCK
+ cmpxchgl %edx, (%ebx)
+ jne 2b
+7: xorl %eax, %eax
+
+9: movl 4(%esp), %esi
+ movl 8(%esp), %ebx
+ addl $12, %esp
+.Ladd_esp:
+ ret
+
+.Lafter_ret:
+1: LOCK
+ incl NWAITERS(%ebx)
+
+.LcleanupSTART:
+6: call __pthread_enable_asynccancel
+ movl %eax, (%esp)
+#if FUTEX_WAIT == 0
+ movl PRIVATE(%ebx), %ecx
+#else
+ movl $FUTEX_WAIT, %ecx
+ orl PRIVATE(%ebx), %ecx
+#endif
+ xorl %esi, %esi
+ xorl %edx, %edx
+ movl $SYS_futex, %eax
+ ENTER_KERNEL
+ movl %eax, %esi
+
+ movl (%esp), %eax
+ call __pthread_disable_asynccancel
+.LcleanupEND:
+
+ testl %esi, %esi
+ je 3f
+ cmpl $-EWOULDBLOCK, %esi
+ jne 4f
+
+3:
+ movl (%ebx), %eax
+5: testl %eax, %eax
+ je 6b
+
+ leal -1(%eax), %edx
+ LOCK
+ cmpxchgl %edx, (%ebx)
+ jne 5b
+
+ LOCK
+ decl NWAITERS(%ebx)
+ jmp 7b
+
+4: LOCK
+ decl NWAITERS(%ebx)
+
+ negl %esi
+#ifdef PIC
+ call __i686.get_pc_thunk.bx
+#else
+ movl $8f, %ebx
+8:
+#endif
+ addl $_GLOBAL_OFFSET_TABLE_, %ebx
+#if USE___THREAD
+# ifdef NO_TLS_DIRECT_SEG_REFS
+ movl errno@gotntpoff(%ebx), %edx
+ addl %gs:0, %edx
+ movl %esi, (%edx)
+# else
+ movl errno@gotntpoff(%ebx), %edx
+ movl %esi, %gs:(%edx)
+# endif
+#else
+ call __errno_location@plt
+ movl %esi, (%eax)
+#endif
+ orl $-1, %eax
+
+ jmp 9b
+ .size __new_sem_wait,.-__new_sem_wait
+ versioned_symbol(libpthread, __new_sem_wait, sem_wait, GLIBC_2_1)
+
+
+ .type sem_wait_cleanup,@function
+sem_wait_cleanup:
+ LOCK
+ decl NWAITERS(%ebx)
+ movl %eax, (%esp)
+.LcallUR:
+ call _Unwind_Resume@PLT
+ hlt
+.LENDCODE:
+ .size sem_wait_cleanup,.-sem_wait_cleanup
+
+
+ .section .gcc_except_table,"a",@progbits
+.LexceptSTART:
+ .byte 0xff # @LPStart format (omit)
+ .byte 0xff # @TType format (omit)
+ .byte 0x01 # call-site format
+ # DW_EH_PE_uleb128
+ .uleb128 .Lcstend-.Lcstbegin
+.Lcstbegin:
+ .uleb128 .LcleanupSTART-.LSTARTCODE
+ .uleb128 .LcleanupEND-.LcleanupSTART
+ .uleb128 sem_wait_cleanup-.LSTARTCODE
+ .uleb128 0
+ .uleb128 .LcallUR-.LSTARTCODE
+ .uleb128 .LENDCODE-.LcallUR
+ .uleb128 0
+ .uleb128 0
+.Lcstend:
+
+
+ .section .eh_frame,"a",@progbits
+.LSTARTFRAME:
+ .long .LENDCIE-.LSTARTCIE # Length of the CIE.
+.LSTARTCIE:
+ .long 0 # CIE ID.
+ .byte 1 # Version number.
+#ifdef SHARED
+ .string "zPLR" # NUL-terminated augmentation
+ # string.
+#else
+ .string "zPL" # NUL-terminated augmentation
+ # string.
+#endif
+ .uleb128 1 # Code alignment factor.
+ .sleb128 -4 # Data alignment factor.
+ .byte 8 # Return address register
+ # column.
+#ifdef SHARED
+ .uleb128 7 # Augmentation value length.
+ .byte 0x9b # Personality: DW_EH_PE_pcrel
+ # + DW_EH_PE_sdata4
+ # + DW_EH_PE_indirect
+ .long DW.ref.__gcc_personality_v0-.
+ .byte 0x1b # LSDA Encoding: DW_EH_PE_pcrel
+ # + DW_EH_PE_sdata4.
+ .byte 0x1b # FDE Encoding: DW_EH_PE_pcrel
+ # + DW_EH_PE_sdata4.
+#else
+ .uleb128 6 # Augmentation value length.
+ .byte 0x0 # Personality: absolute
+ .long __gcc_personality_v0
+ .byte 0x0 # LSDA Encoding: absolute
+#endif
+ .byte 0x0c # DW_CFA_def_cfa
+ .uleb128 4
+ .uleb128 4
+ .byte 0x88 # DW_CFA_offset, column 0x10
+ .uleb128 1
+ .align 4
+.LENDCIE:
+
+ .long .LENDFDE-.LSTARTFDE # Length of the FDE.
+.LSTARTFDE:
+ .long .LSTARTFDE-.LSTARTFRAME # CIE pointer.
+#ifdef SHARED
+ .long .LSTARTCODE-. # PC-relative start address
+ # of the code.
+#else
+ .long .LSTARTCODE # Start address of the code.
+#endif
+ .long .LENDCODE-.LSTARTCODE # Length of the code.
+ .uleb128 4 # Augmentation size
+#ifdef SHARED
+ .long .LexceptSTART-.
+#else
+ .long .LexceptSTART
+#endif
+
+ .byte 4 # DW_CFA_advance_loc4
+ .long .Lpush_ebx-.LSTARTCODE
+ .byte 14 # DW_CFA_def_cfa_offset
+ .uleb128 8
+ .byte 0x83 # DW_CFA_offset %ebx
+ .uleb128 2
+ .byte 4 # DW_CFA_advance_loc4
+ .long .Lpush_esi-.Lpush_ebx
+ .byte 14 # DW_CFA_def_cfa_offset
+ .uleb128 12
+ .byte 0x86 # DW_CFA_offset %esi
+ .uleb128 3
+ .byte 4 # DW_CFA_advance_loc4
+ .long .Lsub_esp-.Lpush_esi
+ .byte 14 # DW_CFA_def_cfa_offset
+ .uleb128 16
+ .byte 4 # DW_CFA_advance_loc4
+ .long .Ladd_esp-.Lsub_esp
+ .byte 14 # DW_CFA_def_cfa_offset
+ .uleb128 4
+ .byte 0xc3 # DW_CFA_restore %ebx
+ .byte 0xc6 # DW_CFA_restore %esi
+ .byte 4 # DW_CFA_advance_loc4
+ .long .Lafter_ret-.Ladd_esp
+ .byte 14 # DW_CFA_def_cfa_offset
+ .uleb128 16
+ .byte 0x83 # DW_CFA_offset %ebx
+ .uleb128 2
+ .byte 0x86 # DW_CFA_offset %esi
+ .uleb128 3
+ .align 4
+.LENDFDE:
+
+
+#ifdef SHARED
+ .hidden DW.ref.__gcc_personality_v0
+ .weak DW.ref.__gcc_personality_v0
+ .section .gnu.linkonce.d.DW.ref.__gcc_personality_v0,"aw",@progbits
+ .align 4
+ .type DW.ref.__gcc_personality_v0, @object
+ .size DW.ref.__gcc_personality_v0, 4
+DW.ref.__gcc_personality_v0:
+ .long __gcc_personality_v0
+#endif
+
+
+#if SHLIB_COMPAT(libpthread, GLIBC_2_0, GLIBC_2_1)
+ .section ".text.compat", "ax"
+ .global __old_sem_wait
+ .type __old_sem_wait,@function
+ .align 16
+ cfi_startproc
+__old_sem_wait:
pushl %ebx
cfi_adjust_cfa_offset(4)
pushl %esi
@@ -52,9 +284,9 @@ __new_sem_wait:
cfi_adjust_cfa_offset(4)
movl 16(%esp), %ebx
- cfi_offset(3, -8) /* %ebx */
+ cfi_offset(ebx, -8)
- cfi_offset(6, -12) /* %esi */
+ cfi_offset(esi, -12)
3: movl (%ebx), %eax
2: testl %eax, %eax
je 1f
@@ -65,17 +297,17 @@ __new_sem_wait:
jne 2b
xorl %eax, %eax
- movl 4(%esp), %esi
- cfi_restore(6)
+5: movl 4(%esp), %esi
movl 8(%esp), %ebx
- cfi_restore(3)
addl $12, %esp
+ cfi_restore(ebx)
+ cfi_restore(esi)
cfi_adjust_cfa_offset(-12)
ret
cfi_adjust_cfa_offset(12)
- cfi_offset(3, -8) /* %ebx */
- cfi_offset(6, -12) /* %esi */
+ cfi_offset(ebx, -8)
+ cfi_offset(esi, -12)
1: call __pthread_enable_asynccancel
movl %eax, (%esp)
@@ -115,25 +347,8 @@ __new_sem_wait:
movl %esi, (%eax)
#endif
orl $-1, %eax
- movl 4(%esp), %esi
- cfi_restore(6)
- movl 8(%esp), %ebx
- cfi_restore(3)
- addl $12, %esp
- cfi_adjust_cfa_offset(-12)
- ret
-
-5: /* Canceled. */
- movl $0xffffffff, %gs:RESULT
- LOCK
- orl $0x10, %gs:CANCELHANDLING
- movl %gs:CLEANUP_JMP_BUF, %eax
- jmp HIDDEN_JUMPTARGET (__pthread_unwind)
+ jmp 5b
cfi_endproc
- .size __new_sem_wait,.-__new_sem_wait
- versioned_symbol(libpthread, __new_sem_wait, sem_wait, GLIBC_2_1)
-#if SHLIB_COMPAT(libpthread, GLIBC_2_0, GLIBC_2_1)
- .global __old_sem_wait
-__old_sem_wait = __new_sem_wait
+ .size __old_sem_wait,.-__old_sem_wait
compat_symbol(libpthread, __old_sem_wait, sem_wait, GLIBC_2_0)
#endif
--- libc/nptl/sysdeps/unix/sysv/linux/ia64/lowlevellock.h 29 Jul 2006 05:06:07 -0000 1.20
+++ libc/nptl/sysdeps/unix/sysv/linux/ia64/lowlevellock.h 13 Aug 2007 18:33:00 -0000 1.26
@@ -25,6 +25,7 @@
#include <bits/pthreadtypes.h>
#include <ia64intrin.h>
#include <atomic.h>
+#include <kernel-features.h>
#define __NR_futex 1230
#define FUTEX_WAIT 0
@@ -36,219 +37,243 @@
#define FUTEX_LOCK_PI 6
#define FUTEX_UNLOCK_PI 7
#define FUTEX_TRYLOCK_PI 8
+#define FUTEX_PRIVATE_FLAG 128
+
+/* Values for 'private' parameter of locking macros. Yes, the
+ definition seems to be backwards. But it is not. The bit will be
+ reversed before passing to the system call. */
+#define LLL_PRIVATE 0
+#define LLL_SHARED FUTEX_PRIVATE_FLAG
+
+
+#if !defined NOT_IN_libc || defined IS_IN_rtld
+/* In libc.so or ld.so all futexes are private. */
+# ifdef __ASSUME_PRIVATE_FUTEX
+# define __lll_private_flag(fl, private) \
+ ((fl) | FUTEX_PRIVATE_FLAG)
+# else
+# define __lll_private_flag(fl, private) \
+ ((fl) | THREAD_GETMEM (THREAD_SELF, header.private_futex))
+# endif
+#else
+# ifdef __ASSUME_PRIVATE_FUTEX
+# define __lll_private_flag(fl, private) \
+ (((fl) | FUTEX_PRIVATE_FLAG) ^ (private))
+# else
+# define __lll_private_flag(fl, private) \
+ (__builtin_constant_p (private) \
+ ? ((private) == 0 \
+ ? ((fl) | THREAD_GETMEM (THREAD_SELF, header.private_futex)) \
+ : (fl)) \
+ : ((fl) | (((private) ^ FUTEX_PRIVATE_FLAG) \
+ & THREAD_GETMEM (THREAD_SELF, header.private_futex))))
+# endif
+#endif
+
/* Delay in spinlock loop. */
#define BUSY_WAIT_NOP asm ("hint @pause")
-/* Initializer for compatibility lock. */
-#define LLL_MUTEX_LOCK_INITIALIZER (0)
+#define lll_futex_wait(futex, val, private) \
+ lll_futex_timed_wait (futex, val, NULL, private)
-#define lll_futex_wait(futex, val) lll_futex_timed_wait (futex, val, 0)
-
-#define lll_futex_timed_wait(ftx, val, timespec) \
+#define lll_futex_timed_wait(ftx, val, timespec, private) \
({ \
- DO_INLINE_SYSCALL(futex, 4, (long) (ftx), FUTEX_WAIT, (int) (val), \
- (long) (timespec)); \
+ DO_INLINE_SYSCALL(futex, 4, (long) (ftx), \
+ __lll_private_flag (FUTEX_WAIT, private), \
+ (int) (val), (long) (timespec)); \
_r10 == -1 ? -_retval : _retval; \
})
-#define lll_futex_wake(ftx, nr) \
+#define lll_futex_wake(ftx, nr, private) \
({ \
- DO_INLINE_SYSCALL(futex, 3, (long) (ftx), FUTEX_WAKE, (int) (nr)); \
+ DO_INLINE_SYSCALL(futex, 3, (long) (ftx), \
+ __lll_private_flag (FUTEX_WAKE, private), \
+ (int) (nr)); \
_r10 == -1 ? -_retval : _retval; \
})
-#define lll_robust_mutex_dead(futexv) \
+#define lll_robust_dead(futexv, private) \
do \
{ \
int *__futexp = &(futexv); \
atomic_or (__futexp, FUTEX_OWNER_DIED); \
- DO_INLINE_SYSCALL(futex, 3, (long) __futexp, FUTEX_WAKE, 1); \
+ DO_INLINE_SYSCALL(futex, 3, (long) __futexp, \
+ __lll_private_flag (FUTEX_WAKE, private), 1); \
} \
while (0)
/* Returns non-zero if error happened, zero if success. */
-#define lll_futex_requeue(ftx, nr_wake, nr_move, mutex, val) \
+#define lll_futex_requeue(ftx, nr_wake, nr_move, mutex, val, private) \
({ \
- DO_INLINE_SYSCALL(futex, 6, (long) (ftx), FUTEX_CMP_REQUEUE, \
+ DO_INLINE_SYSCALL(futex, 6, (long) (ftx), \
+ __lll_private_flag (FUTEX_CMP_REQUEUE, private), \
(int) (nr_wake), (int) (nr_move), (long) (mutex), \
(int) val); \
_r10 == -1; \
})
/* Returns non-zero if error happened, zero if success. */
-#define lll_futex_wake_unlock(ftx, nr_wake, nr_wake2, ftx2) \
+#define lll_futex_wake_unlock(ftx, nr_wake, nr_wake2, ftx2, private) \
({ \
- DO_INLINE_SYSCALL(futex, 6, (long) (ftx), FUTEX_WAKE_OP, \
+ DO_INLINE_SYSCALL(futex, 6, (long) (ftx), \
+ __lll_private_flag (FUTEX_WAKE_OP, private), \
(int) (nr_wake), (int) (nr_wake2), (long) (ftx2), \
FUTEX_OP_CLEAR_WAKE_IF_GT_ONE); \
_r10 == -1; \
})
-#define __lll_mutex_trylock(futex) \
+#define __lll_trylock(futex) \
(atomic_compare_and_exchange_val_acq (futex, 1, 0) != 0)
-#define lll_mutex_trylock(futex) __lll_mutex_trylock (&(futex))
+#define lll_trylock(futex) __lll_trylock (&(futex))
-#define __lll_robust_mutex_trylock(futex, id) \
+#define __lll_robust_trylock(futex, id) \
(atomic_compare_and_exchange_val_acq (futex, id, 0) != 0)
-#define lll_robust_mutex_trylock(futex, id) \
- __lll_robust_mutex_trylock (&(futex), id)
+#define lll_robust_trylock(futex, id) \
+ __lll_robust_trylock (&(futex), id)
-#define __lll_mutex_cond_trylock(futex) \
+#define __lll_cond_trylock(futex) \
(atomic_compare_and_exchange_val_acq (futex, 2, 0) != 0)
-#define lll_mutex_cond_trylock(futex) __lll_mutex_cond_trylock (&(futex))
-
+#define lll_cond_trylock(futex) __lll_cond_trylock (&(futex))
-extern void __lll_lock_wait (int *futex) attribute_hidden;
-extern int __lll_robust_lock_wait (int *futex) attribute_hidden;
-
-#define __lll_mutex_lock(futex) \
- ((void) ({ \
- int *__futex = (futex); \
- if (atomic_compare_and_exchange_bool_acq (__futex, 1, 0) != 0) \
- __lll_lock_wait (__futex); \
+extern void __lll_lock_wait_private (int *futex) attribute_hidden;
+extern void __lll_lock_wait (int *futex, int private) attribute_hidden;
+extern int __lll_robust_lock_wait (int *futex, int private) attribute_hidden;
+
+
+#define __lll_lock(futex, private) \
+ ((void) ({ \
+ int *__futex = (futex); \
+ if (__builtin_expect (atomic_compare_and_exchange_bool_acq (__futex, \
+ 1, 0), 0)) \
+ { \
+ if (__builtin_constant_p (private) && (private) == LLL_PRIVATE) \
+ __lll_lock_wait_private (__futex); \
+ else \
+ __lll_lock_wait (__futex, private); \
+ } \
}))
-#define lll_mutex_lock(futex) __lll_mutex_lock (&(futex))
+#define lll_lock(futex, private) __lll_lock (&(futex), private)
-#define __lll_robust_mutex_lock(futex, id) \
- ({ \
- int *__futex = (futex); \
- int __val = 0; \
- \
- if (atomic_compare_and_exchange_bool_acq (__futex, id, 0) != 0) \
- __val = __lll_robust_lock_wait (__futex); \
- __val; \
+#define __lll_robust_lock(futex, id, private) \
+ ({ \
+ int *__futex = (futex); \
+ int __val = 0; \
+ \
+ if (__builtin_expect (atomic_compare_and_exchange_bool_acq (__futex, id, \
+ 0), 0)) \
+ __val = __lll_robust_lock_wait (__futex, private); \
+ __val; \
})
-#define lll_robust_mutex_lock(futex, id) __lll_robust_mutex_lock (&(futex), id)
+#define lll_robust_lock(futex, id, private) \
+ __lll_robust_lock (&(futex), id, private)
-#define __lll_mutex_cond_lock(futex) \
- ((void) ({ \
- int *__futex = (futex); \
- if (atomic_compare_and_exchange_bool_acq (__futex, 2, 0) != 0) \
- __lll_lock_wait (__futex); \
+#define __lll_cond_lock(futex, private) \
+ ((void) ({ \
+ int *__futex = (futex); \
+ if (__builtin_expect (atomic_compare_and_exchange_bool_acq (__futex, 2, \
+ 0), 0)) \
+ __lll_lock_wait (__futex, private); \
}))
-#define lll_mutex_cond_lock(futex) __lll_mutex_cond_lock (&(futex))
+#define lll_cond_lock(futex, private) __lll_cond_lock (&(futex), private)
-#define __lll_robust_mutex_cond_lock(futex, id) \
- ({ \
- int *__futex = (futex); \
- int __val = 0; \
- int __id = (id) | FUTEX_WAITERS; \
- \
- if (atomic_compare_and_exchange_bool_acq (__futex, __id, 0) != 0) \
- __val = __lll_robust_lock_wait (__futex); \
- __val; \
+#define __lll_robust_cond_lock(futex, id, private) \
+ ({ \
+ int *__futex = (futex); \
+ int __val = 0; \
+ int __id = (id) | FUTEX_WAITERS; \
+ \
+ if (__builtin_expect (atomic_compare_and_exchange_bool_acq (__futex, \
+ __id, 0), 0)) \
+ __val = __lll_robust_lock_wait (__futex, private); \
+ __val; \
})
-#define lll_robust_mutex_cond_lock(futex, id) \
- __lll_robust_mutex_cond_lock (&(futex), id)
+#define lll_robust_cond_lock(futex, id, private) \
+ __lll_robust_cond_lock (&(futex), id, private)
-extern int __lll_timedlock_wait (int *futex, const struct timespec *)
- attribute_hidden;
-extern int __lll_robust_timedlock_wait (int *futex, const struct timespec *)
- attribute_hidden;
-
-
-#define __lll_mutex_timedlock(futex, abstime) \
- ({ \
- int *__futex = (futex); \
- int __val = 0; \
- \
- if (atomic_compare_and_exchange_bool_acq (__futex, 1, 0) != 0) \
- __val = __lll_timedlock_wait (__futex, abstime); \
- __val; \
+extern int __lll_timedlock_wait (int *futex, const struct timespec *,
+ int private) attribute_hidden;
+extern int __lll_robust_timedlock_wait (int *futex, const struct timespec *,
+ int private) attribute_hidden;
+
+
+#define __lll_timedlock(futex, abstime, private) \
+ ({ \
+ int *__futex = (futex); \
+ int __val = 0; \
+ \
+ if (__builtin_expect (atomic_compare_and_exchange_bool_acq (__futex, 1, \
+ 0), 0)) \
+ __val = __lll_timedlock_wait (__futex, abstime, private); \
+ __val; \
})
-#define lll_mutex_timedlock(futex, abstime) \
- __lll_mutex_timedlock (&(futex), abstime)
+#define lll_timedlock(futex, abstime, private) \
+ __lll_timedlock (&(futex), abstime, private)
-#define __lll_robust_mutex_timedlock(futex, abstime, id) \
- ({ \
- int *__futex = (futex); \
- int __val = 0; \
- \
- if (atomic_compare_and_exchange_bool_acq (__futex, id, 0) != 0) \
- __val = __lll_robust_timedlock_wait (__futex, abstime); \
- __val; \
+#define __lll_robust_timedlock(futex, abstime, id, private) \
+ ({ \
+ int *__futex = (futex); \
+ int __val = 0; \
+ \
+ if (__builtin_expect (atomic_compare_and_exchange_bool_acq (__futex, id, \
+ 0), 0)) \
+ __val = __lll_robust_timedlock_wait (__futex, abstime, private); \
+ __val; \
})
-#define lll_robust_mutex_timedlock(futex, abstime, id) \
- __lll_robust_mutex_timedlock (&(futex), abstime, id)
-
-
-#define __lll_mutex_unlock(futex) \
- ((void) ({ \
- int *__futex = (futex); \
- int __val = atomic_exchange_rel (__futex, 0); \
- \
- if (__builtin_expect (__val > 1, 0)) \
- lll_futex_wake (__futex, 1); \
- }))
-#define lll_mutex_unlock(futex) \
- __lll_mutex_unlock(&(futex))
+#define lll_robust_timedlock(futex, abstime, id, private) \
+ __lll_robust_timedlock (&(futex), abstime, id, private)
-#define __lll_robust_mutex_unlock(futex) \
- ((void) ({ \
- int *__futex = (futex); \
- int __val = atomic_exchange_rel (__futex, 0); \
- \
- if (__builtin_expect (__val & FUTEX_WAITERS, 0)) \
- lll_futex_wake (__futex, 1); \
+#define __lll_unlock(futex, private) \
+ ((void) ({ \
+ int *__futex = (futex); \
+ int __val = atomic_exchange_rel (__futex, 0); \
+ \
+ if (__builtin_expect (__val > 1, 0)) \
+ lll_futex_wake (__futex, 1, private); \
}))
-#define lll_robust_mutex_unlock(futex) \
- __lll_robust_mutex_unlock(&(futex))
+#define lll_unlock(futex, private) __lll_unlock(&(futex), private)
-#define __lll_mutex_unlock_force(futex) \
- ((void) ({ \
- int *__futex = (futex); \
- (void) atomic_exchange_rel (__futex, 0); \
- lll_futex_wake (__futex, 1); \
+#define __lll_robust_unlock(futex, private) \
+ ((void) ({ \
+ int *__futex = (futex); \
+ int __val = atomic_exchange_rel (__futex, 0); \
+ \
+ if (__builtin_expect (__val & FUTEX_WAITERS, 0)) \
+ lll_futex_wake (__futex, 1, private); \
}))
-#define lll_mutex_unlock_force(futex) \
- __lll_mutex_unlock_force(&(futex))
+#define lll_robust_unlock(futex, private) \
+ __lll_robust_unlock(&(futex), private)
-#define lll_mutex_islocked(futex) \
+#define lll_islocked(futex) \
(futex != 0)
-
-/* We have a separate internal lock implementation which is not tied
- to binary compatibility. We can use the lll_mutex_*. */
-
-/* Type for lock object. */
-typedef int lll_lock_t;
-
-extern int lll_unlock_wake_cb (int *__futex) attribute_hidden;
-
/* Initializers for lock. */
#define LLL_LOCK_INITIALIZER (0)
#define LLL_LOCK_INITIALIZER_LOCKED (1)
-#define lll_trylock(futex) lll_mutex_trylock (futex)
-#define lll_lock(futex) lll_mutex_lock (futex)
-#define lll_unlock(futex) lll_mutex_unlock (futex)
-#define lll_islocked(futex) lll_mutex_islocked (futex)
-
-
/* The kernel notifies a process with uses CLONE_CLEARTID via futex
wakeup when the clone terminates. The memory location contains the
thread ID while the clone is running and is reset to zero
afterwards. */
#define lll_wait_tid(tid) \
- do \
- { \
- __typeof (tid) __tid; \
- while ((__tid = (tid)) != 0) \
- lll_futex_wait (&(tid), __tid); \
- } \
+ do \
+ { \
+ __typeof (tid) __tid; \
+ while ((__tid = (tid)) != 0) \
+ lll_futex_wait (&(tid), __tid, LLL_SHARED); \
+ } \
while (0)
extern int __lll_timedwait_tid (int *, const struct timespec *)
@@ -262,26 +287,4 @@ extern int __lll_timedwait_tid (int *, c
__res; \
})
-
-/* Conditional variable handling. */
-
-extern void __lll_cond_wait (pthread_cond_t *cond)
- attribute_hidden;
-extern int __lll_cond_timedwait (pthread_cond_t *cond,
- const struct timespec *abstime)
- attribute_hidden;
-extern void __lll_cond_wake (pthread_cond_t *cond)
- attribute_hidden;
-extern void __lll_cond_broadcast (pthread_cond_t *cond)
- attribute_hidden;
-
-#define lll_cond_wait(cond) \
- __lll_cond_wait (cond)
-#define lll_cond_timedwait(cond, abstime) \
- __lll_cond_timedwait (cond, abstime)
-#define lll_cond_wake(cond) \
- __lll_cond_wake (cond)
-#define lll_cond_broadcast(cond) \
- __lll_cond_broadcast (cond)
-
#endif /* lowlevellock.h */
--- libc/nptl/sysdeps/unix/sysv/linux/ia64/pthread_once.c 8 Sep 2004 06:10:01 -0000 1.6
+++ libc/nptl/sysdeps/unix/sysv/linux/ia64/pthread_once.c 28 Jul 2007 19:31:59 -0000 1.7
@@ -30,7 +30,7 @@ clear_once_control (void *arg)
pthread_once_t *once_control = (pthread_once_t *) arg;
*once_control = 0;
- lll_futex_wake (once_control, INT_MAX);
+ lll_futex_wake (once_control, INT_MAX, LLL_PRIVATE);
}
@@ -65,7 +65,7 @@ __pthread_once (once_control, init_routi
if (((oldval ^ newval) & -4) == 0)
{
/* Same generation, some other thread was faster. Wait. */
- lll_futex_wait (once_control, newval);
+ lll_futex_wait (once_control, newval, LLL_PRIVATE);
continue;
}
}
@@ -84,7 +84,7 @@ __pthread_once (once_control, init_routi
atomic_increment (once_control);
/* Wake up all other threads. */
- lll_futex_wake (once_control, INT_MAX);
+ lll_futex_wake (once_control, INT_MAX, LLL_PRIVATE);
break;
}
--- libc/nptl/sysdeps/unix/sysv/linux/ia64/bits/pthreadtypes.h 15 Feb 2006 17:13:30 -0000 1.17
+++ libc/nptl/sysdeps/unix/sysv/linux/ia64/bits/pthreadtypes.h 12 Aug 2007 19:39:31 -0000 1.18
@@ -126,9 +126,9 @@ typedef union
unsigned int __nr_readers_queued;
unsigned int __nr_writers_queued;
int __writer;
- int __pad1;
+ int __shared;
+ unsigned long int __pad1;
unsigned long int __pad2;
- unsigned long int __pad3;
/* FLAGS must stay at this position in the structure to maintain
binary compatibility. */
unsigned int __flags;
--- libc/nptl/sysdeps/unix/sysv/linux/powerpc/lowlevellock.h 26 Mar 2007 20:12:10 -0000 1.24
+++ libc/nptl/sysdeps/unix/sysv/linux/powerpc/lowlevellock.h 13 Aug 2007 18:33:00 -0000 1.32
@@ -24,6 +24,7 @@
#include <sys/param.h>
#include <bits/pthreadtypes.h>
#include <atomic.h>
+#include <kernel-features.h>
#ifndef __NR_futex
@@ -38,76 +39,101 @@
#define FUTEX_LOCK_PI 6
#define FUTEX_UNLOCK_PI 7
#define FUTEX_TRYLOCK_PI 8
+#define FUTEX_PRIVATE_FLAG 128
-/* Initializer for compatibility lock. */
-#define LLL_MUTEX_LOCK_INITIALIZER (0)
+/* Values for 'private' parameter of locking macros. Yes, the
+ definition seems to be backwards. But it is not. The bit will be
+ reversed before passing to the system call. */
+#define LLL_PRIVATE 0
+#define LLL_SHARED FUTEX_PRIVATE_FLAG
+
+#if !defined NOT_IN_libc || defined IS_IN_rtld
+/* In libc.so or ld.so all futexes are private. */
+# ifdef __ASSUME_PRIVATE_FUTEX
+# define __lll_private_flag(fl, private) \
+ ((fl) | FUTEX_PRIVATE_FLAG)
+# else
+# define __lll_private_flag(fl, private) \
+ ((fl) | THREAD_GETMEM (THREAD_SELF, header.private_futex))
+# endif
+#else
+# ifdef __ASSUME_PRIVATE_FUTEX
+# define __lll_private_flag(fl, private) \
+ (((fl) | FUTEX_PRIVATE_FLAG) ^ (private))
+# else
+# define __lll_private_flag(fl, private) \
+ (__builtin_constant_p (private) \
+ ? ((private) == 0 \
+ ? ((fl) | THREAD_GETMEM (THREAD_SELF, header.private_futex)) \
+ : (fl)) \
+ : ((fl) | (((private) ^ FUTEX_PRIVATE_FLAG) \
+ & THREAD_GETMEM (THREAD_SELF, header.private_futex))))
+# endif
+#endif
-#define lll_futex_wait(futexp, val) \
- ({ \
- INTERNAL_SYSCALL_DECL (__err); \
- long int __ret; \
- \
- __ret = INTERNAL_SYSCALL (futex, __err, 4, \
- (futexp), FUTEX_WAIT, (val), 0); \
- INTERNAL_SYSCALL_ERROR_P (__ret, __err) ? -__ret : __ret; \
- })
+#define lll_futex_wait(futexp, val, private) \
+ lll_futex_timed_wait (futexp, val, NULL, private)
-#define lll_futex_timed_wait(futexp, val, timespec) \
+#define lll_futex_timed_wait(futexp, val, timespec, private) \
({ \
INTERNAL_SYSCALL_DECL (__err); \
long int __ret; \
\
- __ret = INTERNAL_SYSCALL (futex, __err, 4, \
- (futexp), FUTEX_WAIT, (val), (timespec)); \
+ __ret = INTERNAL_SYSCALL (futex, __err, 4, (futexp), \
+ __lll_private_flag (FUTEX_WAIT, private), \
+ (val), (timespec)); \
INTERNAL_SYSCALL_ERROR_P (__ret, __err) ? -__ret : __ret; \
})
-#define lll_futex_wake(futexp, nr) \
+#define lll_futex_wake(futexp, nr, private) \
({ \
INTERNAL_SYSCALL_DECL (__err); \
long int __ret; \
\
- __ret = INTERNAL_SYSCALL (futex, __err, 4, \
- (futexp), FUTEX_WAKE, (nr), 0); \
+ __ret = INTERNAL_SYSCALL (futex, __err, 4, (futexp), \
+ __lll_private_flag (FUTEX_WAKE, private), \
+ (nr), 0); \
INTERNAL_SYSCALL_ERROR_P (__ret, __err) ? -__ret : __ret; \
})
-#define lll_robust_mutex_dead(futexv) \
+#define lll_robust_dead(futexv, private) \
do \
{ \
INTERNAL_SYSCALL_DECL (__err); \
int *__futexp = &(futexv); \
\
atomic_or (__futexp, FUTEX_OWNER_DIED); \
- INTERNAL_SYSCALL (futex, __err, 4, __futexp, FUTEX_WAKE, 1, 0); \
+ INTERNAL_SYSCALL (futex, __err, 4, __futexp, \
+ __lll_private_flag (FUTEX_WAKE, private), 1, 0); \
} \
while (0)
/* Returns non-zero if error happened, zero if success. */
-#define lll_futex_requeue(futexp, nr_wake, nr_move, mutex, val) \
+#define lll_futex_requeue(futexp, nr_wake, nr_move, mutex, val, private) \
({ \
INTERNAL_SYSCALL_DECL (__err); \
long int __ret; \
\
- __ret = INTERNAL_SYSCALL (futex, __err, 6, \
- (futexp), FUTEX_CMP_REQUEUE, (nr_wake), \
- (nr_move), (mutex), (val)); \
+ __ret = INTERNAL_SYSCALL (futex, __err, 6, (futexp), \
+ __lll_private_flag (FUTEX_CMP_REQUEUE, private),\
+ (nr_wake), (nr_move), (mutex), (val)); \
INTERNAL_SYSCALL_ERROR_P (__ret, __err); \
})
/* Returns non-zero if error happened, zero if success. */
-#define lll_futex_wake_unlock(futexp, nr_wake, nr_wake2, futexp2) \
+#define lll_futex_wake_unlock(futexp, nr_wake, nr_wake2, futexp2, private) \
({ \
INTERNAL_SYSCALL_DECL (__err); \
long int __ret; \
\
- __ret = INTERNAL_SYSCALL (futex, __err, 6, \
- (futexp), FUTEX_WAKE_OP, (nr_wake), \
- (nr_wake2), (futexp2), \
+ __ret = INTERNAL_SYSCALL (futex, __err, 6, (futexp), \
+ __lll_private_flag (FUTEX_WAKE_OP, private), \
+ (nr_wake), (nr_wake2), (futexp2), \
FUTEX_OP_CLEAR_WAKE_IF_GT_ONE); \
INTERNAL_SYSCALL_ERROR_P (__ret, __err); \
})
-
+
+
#ifdef UP
# define __lll_acq_instr ""
# define __lll_rel_instr ""
@@ -145,135 +171,120 @@
__val; \
})
-#define lll_robust_mutex_trylock(lock, id) __lll_robust_trylock (&(lock), id)
+#define lll_robust_trylock(lock, id) __lll_robust_trylock (&(lock), id)
/* Set *futex to 1 if it is 0, atomically. Returns the old value */
#define __lll_trylock(futex) __lll_robust_trylock (futex, 1)
-#define lll_mutex_trylock(lock) __lll_trylock (&(lock))
+#define lll_trylock(lock) __lll_trylock (&(lock))
/* Set *futex to 2 if it is 0, atomically. Returns the old value */
#define __lll_cond_trylock(futex) __lll_robust_trylock (futex, 2)
-#define lll_mutex_cond_trylock(lock) __lll_cond_trylock (&(lock))
+#define lll_cond_trylock(lock) __lll_cond_trylock (&(lock))
-extern void __lll_lock_wait (int *futex) attribute_hidden;
-extern int __lll_robust_lock_wait (int *futex) attribute_hidden;
+extern void __lll_lock_wait_private (int *futex) attribute_hidden;
+extern void __lll_lock_wait (int *futex, int private) attribute_hidden;
+extern int __lll_robust_lock_wait (int *futex, int private) attribute_hidden;
-#define lll_mutex_lock(lock) \
+#define lll_lock(lock, private) \
(void) ({ \
int *__futex = &(lock); \
if (__builtin_expect (atomic_compare_and_exchange_val_acq (__futex, 1, 0),\
0) != 0) \
- __lll_lock_wait (__futex); \
+ { \
+ if (__builtin_constant_p (private) && (private) == LLL_PRIVATE) \
+ __lll_lock_wait_private (__futex); \
+ else \
+ __lll_lock_wait (__futex, private); \
+ } \
})
-#define lll_robust_mutex_lock(lock, id) \
+#define lll_robust_lock(lock, id, private) \
({ \
int *__futex = &(lock); \
int __val = 0; \
if (__builtin_expect (atomic_compare_and_exchange_bool_acq (__futex, id, \
0), 0)) \
- __val = __lll_robust_lock_wait (__futex); \
+ __val = __lll_robust_lock_wait (__futex, private); \
__val; \
})
-#define lll_mutex_cond_lock(lock) \
+#define lll_cond_lock(lock, private) \
(void) ({ \
int *__futex = &(lock); \
if (__builtin_expect (atomic_compare_and_exchange_val_acq (__futex, 2, 0),\
0) != 0) \
- __lll_lock_wait (__futex); \
+ __lll_lock_wait (__futex, private); \
})
-#define lll_robust_mutex_cond_lock(lock, id) \
+#define lll_robust_cond_lock(lock, id, private) \
({ \
int *__futex = &(lock); \
int __val = 0; \
int __id = id | FUTEX_WAITERS; \
if (__builtin_expect (atomic_compare_and_exchange_bool_acq (__futex, __id,\
0), 0)) \
- __val = __lll_robust_lock_wait (__futex); \
+ __val = __lll_robust_lock_wait (__futex, private); \
__val; \
})
extern int __lll_timedlock_wait
- (int *futex, const struct timespec *) attribute_hidden;
+ (int *futex, const struct timespec *, int private) attribute_hidden;
extern int __lll_robust_timedlock_wait
- (int *futex, const struct timespec *) attribute_hidden;
+ (int *futex, const struct timespec *, int private) attribute_hidden;
-#define lll_mutex_timedlock(lock, abstime) \
+#define lll_timedlock(lock, abstime, private) \
({ \
int *__futex = &(lock); \
int __val = 0; \
if (__builtin_expect (atomic_compare_and_exchange_val_acq (__futex, 1, 0),\
0) != 0) \
- __val = __lll_timedlock_wait (__futex, abstime); \
+ __val = __lll_timedlock_wait (__futex, abstime, private); \
__val; \
})
-#define lll_robust_mutex_timedlock(lock, abstime, id) \
+#define lll_robust_timedlock(lock, abstime, id, private) \
({ \
int *__futex = &(lock); \
int __val = 0; \
if (__builtin_expect (atomic_compare_and_exchange_bool_acq (__futex, id, \
0), 0)) \
- __val = __lll_robust_timedlock_wait (__futex, abstime); \
+ __val = __lll_robust_timedlock_wait (__futex, abstime, private); \
__val; \
})
-#define lll_mutex_unlock(lock) \
+#define lll_unlock(lock, private) \
((void) ({ \
int *__futex = &(lock); \
int __val = atomic_exchange_rel (__futex, 0); \
if (__builtin_expect (__val > 1, 0)) \
- lll_futex_wake (__futex, 1); \
+ lll_futex_wake (__futex, 1, private); \
}))
-#define lll_robust_mutex_unlock(lock) \
+#define lll_robust_unlock(lock, private) \
((void) ({ \
int *__futex = &(lock); \
int __val = atomic_exchange_rel (__futex, 0); \
if (__builtin_expect (__val & FUTEX_WAITERS, 0)) \
- lll_futex_wake (__futex, 1); \
- }))
-
-#define lll_mutex_unlock_force(lock) \
- ((void) ({ \
- int *__futex = &(lock); \
- *__futex = 0; \
- __asm __volatile (__lll_rel_instr ::: "memory"); \
- lll_futex_wake (__futex, 1); \
+ lll_futex_wake (__futex, 1, private); \
}))
-#define lll_mutex_islocked(futex) \
+#define lll_islocked(futex) \
(futex != 0)
-/* Our internal lock implementation is identical to the binary-compatible
- mutex implementation. */
-
-/* Type for lock object. */
-typedef int lll_lock_t;
-
/* Initializers for lock. */
#define LLL_LOCK_INITIALIZER (0)
#define LLL_LOCK_INITIALIZER_LOCKED (1)
-extern int lll_unlock_wake_cb (int *__futex) attribute_hidden;
-
/* The states of a lock are:
0 - untaken
1 - taken by one user
>1 - taken by more users */
-#define lll_trylock(lock) lll_mutex_trylock (lock)
-#define lll_lock(lock) lll_mutex_lock (lock)
-#define lll_unlock(lock) lll_mutex_unlock (lock)
-#define lll_islocked(lock) lll_mutex_islocked (lock)
-
/* The kernel notifies a process which uses CLONE_CLEARTID via futex
wakeup when the clone terminates. The memory location contains the
thread ID while the clone is running and is reset to zero
@@ -282,7 +293,7 @@ extern int lll_unlock_wake_cb (int *__fu
do { \
__typeof (tid) __tid; \
while ((__tid = (tid)) != 0) \
- lll_futex_wait (&(tid), __tid); \
+ lll_futex_wait (&(tid), __tid, LLL_SHARED); \
} while (0)
extern int __lll_timedwait_tid (int *, const struct timespec *)
@@ -296,26 +307,4 @@ extern int __lll_timedwait_tid (int *, c
__res; \
})
-
-/* Conditional variable handling. */
-
-extern void __lll_cond_wait (pthread_cond_t *cond)
- attribute_hidden;
-extern int __lll_cond_timedwait (pthread_cond_t *cond,
- const struct timespec *abstime)
- attribute_hidden;
-extern void __lll_cond_wake (pthread_cond_t *cond)
- attribute_hidden;
-extern void __lll_cond_broadcast (pthread_cond_t *cond)
- attribute_hidden;
-
-#define lll_cond_wait(cond) \
- __lll_cond_wait (cond)
-#define lll_cond_timedwait(cond, abstime) \
- __lll_cond_timedwait (cond, abstime)
-#define lll_cond_wake(cond) \
- __lll_cond_wake (cond)
-#define lll_cond_broadcast(cond) \
- __lll_cond_broadcast (cond)
-
#endif /* lowlevellock.h */
--- libc/nptl/sysdeps/unix/sysv/linux/powerpc/pthread_once.c 8 Sep 2004 06:07:35 -0000 1.4
+++ libc/nptl/sysdeps/unix/sysv/linux/powerpc/pthread_once.c 28 Jul 2007 19:30:58 -0000 1.6
@@ -30,7 +30,7 @@ clear_once_control (void *arg)
pthread_once_t *once_control = (pthread_once_t *) arg;
*once_control = 0;
- lll_futex_wake (once_control, INT_MAX);
+ lll_futex_wake (once_control, INT_MAX, LLL_PRIVATE);
}
@@ -74,7 +74,7 @@ __pthread_once (pthread_once_t *once_con
break;
/* Same generation, some other thread was faster. Wait. */
- lll_futex_wait (once_control, oldval);
+ lll_futex_wait (once_control, oldval, LLL_PRIVATE);
}
@@ -92,7 +92,7 @@ __pthread_once (pthread_once_t *once_con
atomic_increment (once_control);
/* Wake up all other threads. */
- lll_futex_wake (once_control, INT_MAX);
+ lll_futex_wake (once_control, INT_MAX, LLL_PRIVATE);
return 0;
}
--- libc/nptl/sysdeps/unix/sysv/linux/powerpc/sem_post.c 15 May 2007 06:31:57 -0000 1.4
+++ libc/nptl/sysdeps/unix/sysv/linux/powerpc/sem_post.c 1 Aug 2007 04:34:01 -0000 1.6
@@ -29,11 +29,37 @@
int
__new_sem_post (sem_t *sem)
{
+ struct new_sem *isem = (struct new_sem *) sem;
+
+ __asm __volatile (__lll_rel_instr ::: "memory");
+ atomic_increment (&isem->value);
+ __asm __volatile (__lll_acq_instr ::: "memory");
+ if (isem->nwaiters > 0)
+ {
+ int err = lll_futex_wake (&isem->value, 1,
+ isem->private ^ FUTEX_PRIVATE_FLAG);
+ if (__builtin_expect (err, 0) < 0)
+ {
+ __set_errno (-err);
+ return -1;
+ }
+ }
+ return 0;
+}
+versioned_symbol (libpthread, __new_sem_post, sem_post, GLIBC_2_1);
+
+#if SHLIB_COMPAT (libpthread, GLIBC_2_0, GLIBC_2_1)
+
+int
+attribute_compat_text_section
+__old_sem_post (sem_t *sem)
+{
int *futex = (int *) sem;
__asm __volatile (__lll_rel_instr ::: "memory");
int nr = atomic_increment_val (futex);
- int err = lll_futex_wake (futex, nr);
+ /* We always have to assume it is a shared semaphore. */
+ int err = lll_futex_wake (futex, 1, LLL_SHARED);
if (__builtin_expect (err, 0) < 0)
{
__set_errno (-err);
@@ -41,8 +67,6 @@ __new_sem_post (sem_t *sem)
}
return 0;
}
-versioned_symbol (libpthread, __new_sem_post, sem_post, GLIBC_2_1);
-#if SHLIB_COMPAT (libpthread, GLIBC_2_0, GLIBC_2_1)
-strong_alias (__new_sem_post, __old_sem_post)
+
compat_symbol (libpthread, __old_sem_post, sem_post, GLIBC_2_0);
#endif
--- libc/nptl/sysdeps/unix/sysv/linux/powerpc/bits/pthreadtypes.h 15 Feb 2006 17:14:33 -0000 1.19
+++ libc/nptl/sysdeps/unix/sysv/linux/powerpc/bits/pthreadtypes.h 27 May 2007 18:02:09 -0000 1.20
@@ -160,9 +160,9 @@ typedef union
unsigned int __nr_readers_queued;
unsigned int __nr_writers_queued;
int __writer;
- int __pad1;
+ int __shared;
+ unsigned long int __pad1;
unsigned long int __pad2;
- unsigned long int __pad3;
/* FLAGS must stay at this position in the structure to maintain
binary compatibility. */
unsigned int __flags;
@@ -176,9 +176,12 @@ typedef union
unsigned int __writer_wakeup;
unsigned int __nr_readers_queued;
unsigned int __nr_writers_queued;
+ unsigned char __pad1;
+ unsigned char __pad2;
+ unsigned char __shared;
/* FLAGS must stay at this position in the structure to maintain
binary compatibility. */
- unsigned int __flags;
+ unsigned char __flags;
int __writer;
} __data;
# endif
--- libc/nptl/sysdeps/unix/sysv/linux/s390/lowlevellock.h 29 Jul 2006 05:06:07 -0000 1.15
+++ libc/nptl/sysdeps/unix/sysv/linux/s390/lowlevellock.h 14 Aug 2007 19:37:50 -0000 1.21
@@ -24,6 +24,7 @@
#include <sys/param.h>
#include <bits/pthreadtypes.h>
#include <atomic.h>
+#include <kernel-features.h>
#define SYS_futex 238
#define FUTEX_WAIT 0
@@ -35,31 +36,47 @@
#define FUTEX_LOCK_PI 6
#define FUTEX_UNLOCK_PI 7
#define FUTEX_TRYLOCK_PI 8
+#define FUTEX_PRIVATE_FLAG 128
-/* Initializer for compatibility lock. */
-#define LLL_MUTEX_LOCK_INITIALIZER (0)
-
-#define lll_futex_wait(futex, val) \
- ({ \
- register unsigned long int __r2 asm ("2") = (unsigned long int) (futex); \
- register unsigned long int __r3 asm ("3") = FUTEX_WAIT; \
- register unsigned long int __r4 asm ("4") = (unsigned long int) (val); \
- register unsigned long int __r5 asm ("5") = 0ul; \
- register unsigned long __result asm ("2"); \
- \
- __asm __volatile ("svc %b1" \
- : "=d" (__result) \
- : "i" (SYS_futex), "0" (__r2), "d" (__r3), \
- "d" (__r4), "d" (__r5) \
- : "cc", "memory" ); \
- __result; \
- })
+/* Values for 'private' parameter of locking macros. Yes, the
+ definition seems to be backwards. But it is not. The bit will be
+ reversed before passing to the system call. */
+#define LLL_PRIVATE 0
+#define LLL_SHARED FUTEX_PRIVATE_FLAG
+
+
+#if !defined NOT_IN_libc || defined IS_IN_rtld
+/* In libc.so or ld.so all futexes are private. */
+# ifdef __ASSUME_PRIVATE_FUTEX
+# define __lll_private_flag(fl, private) \
+ ((fl) | FUTEX_PRIVATE_FLAG)
+# else
+# define __lll_private_flag(fl, private) \
+ ((fl) | THREAD_GETMEM (THREAD_SELF, header.private_futex))
+# endif
+#else
+# ifdef __ASSUME_PRIVATE_FUTEX
+# define __lll_private_flag(fl, private) \
+ (((fl) | FUTEX_PRIVATE_FLAG) ^ (private))
+# else
+# define __lll_private_flag(fl, private) \
+ (__builtin_constant_p (private) \
+ ? ((private) == 0 \
+ ? ((fl) | THREAD_GETMEM (THREAD_SELF, header.private_futex)) \
+ : (fl)) \
+ : ((fl) | (((private) ^ FUTEX_PRIVATE_FLAG) \
+ & THREAD_GETMEM (THREAD_SELF, header.private_futex))))
+# endif
+#endif
+#define lll_futex_wait(futex, val, private) \
+ lll_futex_timed_wait (futex, val, NULL, private)
-#define lll_futex_timed_wait(futex, val, timespec) \
+#define lll_futex_timed_wait(futex, val, timespec, private) \
({ \
register unsigned long int __r2 asm ("2") = (unsigned long int) (futex); \
- register unsigned long int __r3 asm ("3") = FUTEX_WAIT; \
+ register unsigned long int __r3 asm ("3") \
+ = __lll_private_flag (FUTEX_WAIT, private); \
register unsigned long int __r4 asm ("4") = (unsigned long int) (val); \
register unsigned long int __r5 asm ("5") = (unsigned long int)(timespec);\
register unsigned long int __result asm ("2"); \
@@ -73,10 +90,11 @@
})
-#define lll_futex_wake(futex, nr) \
+#define lll_futex_wake(futex, nr, private) \
({ \
register unsigned long int __r2 asm ("2") = (unsigned long int) (futex); \
- register unsigned long int __r3 asm ("3") = FUTEX_WAKE; \
+ register unsigned long int __r3 asm ("3") \
+ = __lll_private_flag (FUTEX_WAKE, private); \
register unsigned long int __r4 asm ("4") = (unsigned long int) (nr); \
register unsigned long int __result asm ("2"); \
\
@@ -88,22 +106,23 @@
})
-#define lll_robust_mutex_dead(futexv) \
+#define lll_robust_dead(futexv, private) \
do \
{ \
int *__futexp = &(futexv); \
\
atomic_or (__futexp, FUTEX_OWNER_DIED); \
- lll_futex_wake (__futexp, 1); \
+ lll_futex_wake (__futexp, 1, private); \
} \
while (0)
/* Returns non-zero if error happened, zero if success. */
-#define lll_futex_requeue(futex, nr_wake, nr_move, mutex, val) \
+#define lll_futex_requeue(futex, nr_wake, nr_move, mutex, val, private) \
({ \
register unsigned long int __r2 asm ("2") = (unsigned long int) (futex); \
- register unsigned long int __r3 asm ("3") = FUTEX_CMP_REQUEUE; \
+ register unsigned long int __r3 asm ("3") \
+ = __lll_private_flag (FUTEX_CMP_REQUEUE, private); \
register unsigned long int __r4 asm ("4") = (long int) (nr_wake); \
register unsigned long int __r5 asm ("5") = (long int) (nr_move); \
register unsigned long int __r6 asm ("6") = (unsigned long int) (mutex); \
@@ -120,10 +139,11 @@
/* Returns non-zero if error happened, zero if success. */
-#define lll_futex_wake_unlock(futex, nr_wake, nr_wake2, futex2) \
+#define lll_futex_wake_unlock(futex, nr_wake, nr_wake2, futex2, private) \
({ \
register unsigned long int __r2 asm ("2") = (unsigned long int) (futex); \
- register unsigned long int __r3 asm ("3") = FUTEX_WAKE_OP; \
+ register unsigned long int __r3 asm ("3") \
+ = __lll_private_flag (FUTEX_WAKE_OP, private); \
register unsigned long int __r4 asm ("4") = (long int) (nr_wake); \
register unsigned long int __r5 asm ("5") = (long int) (nr_wake2); \
register unsigned long int __r6 asm ("6") = (unsigned long int) (futex2); \
@@ -155,7 +175,7 @@
static inline int
__attribute__ ((always_inline))
-__lll_mutex_trylock (int *futex)
+__lll_trylock (int *futex)
{
unsigned int old;
@@ -164,12 +184,12 @@ __lll_mutex_trylock (int *futex)
: "0" (0), "d" (1), "m" (*futex) : "cc", "memory" );
return old != 0;
}
-#define lll_mutex_trylock(futex) __lll_mutex_trylock (&(futex))
+#define lll_trylock(futex) __lll_trylock (&(futex))
static inline int
__attribute__ ((always_inline))
-__lll_mutex_cond_trylock (int *futex)
+__lll_cond_trylock (int *futex)
{
unsigned int old;
@@ -178,12 +198,12 @@ __lll_mutex_cond_trylock (int *futex)
: "0" (0), "d" (2), "m" (*futex) : "cc", "memory" );
return old != 0;
}
-#define lll_mutex_cond_trylock(futex) __lll_mutex_cond_trylock (&(futex))
+#define lll_cond_trylock(futex) __lll_cond_trylock (&(futex))
static inline int
__attribute__ ((always_inline))
-__lll_robust_mutex_trylock (int *futex, int id)
+__lll_robust_trylock (int *futex, int id)
{
unsigned int old;
@@ -192,156 +212,132 @@ __lll_robust_mutex_trylock (int *futex,
: "0" (0), "d" (id), "m" (*futex) : "cc", "memory" );
return old != 0;
}
-#define lll_robust_mutex_trylock(futex, id) \
- __lll_robust_mutex_trylock (&(futex), id)
+#define lll_robust_trylock(futex, id) \
+ __lll_robust_trylock (&(futex), id)
-extern void __lll_lock_wait (int *futex) attribute_hidden;
-extern int __lll_robust_lock_wait (int *futex) attribute_hidden;
+extern void __lll_lock_wait_private (int *futex) attribute_hidden;
+extern void __lll_lock_wait (int *futex, int private) attribute_hidden;
+extern int __lll_robust_lock_wait (int *futex, int private) attribute_hidden;
static inline void
__attribute__ ((always_inline))
-__lll_mutex_lock (int *futex)
+__lll_lock (int *futex, int private)
{
- if (atomic_compare_and_exchange_bool_acq (futex, 1, 0) != 0)
- __lll_lock_wait (futex);
+ if (__builtin_expect (atomic_compare_and_exchange_bool_acq (futex, 1, 0), 0))
+ {
+ if (__builtin_constant_p (private) && private == LLL_PRIVATE)
+ __lll_lock_wait_private (futex);
+ else
+ __lll_lock_wait (futex, private);
+ }
}
-#define lll_mutex_lock(futex) __lll_mutex_lock (&(futex))
+#define lll_lock(futex, private) __lll_lock (&(futex), private)
static inline int
__attribute__ ((always_inline))
-__lll_robust_mutex_lock (int *futex, int id)
+__lll_robust_lock (int *futex, int id, int private)
{
int result = 0;
- if (atomic_compare_and_exchange_bool_acq (futex, id, 0) != 0)
- result = __lll_robust_lock_wait (futex);
+ if (__builtin_expect (atomic_compare_and_exchange_bool_acq (futex, id, 0),
+ 0))
+ result = __lll_robust_lock_wait (futex, private);
return result;
}
-#define lll_robust_mutex_lock(futex, id) __lll_robust_mutex_lock (&(futex), id)
+#define lll_robust_lock(futex, id, private) \
+ __lll_robust_lock (&(futex), id, private)
static inline void
__attribute__ ((always_inline))
-__lll_mutex_cond_lock (int *futex)
+__lll_cond_lock (int *futex, int private)
{
- if (atomic_compare_and_exchange_bool_acq (futex, 2, 0) != 0)
- __lll_lock_wait (futex);
+ if (__builtin_expect (atomic_compare_and_exchange_bool_acq (futex, 2, 0), 0))
+ __lll_lock_wait (futex, private);
}
-#define lll_mutex_cond_lock(futex) __lll_mutex_cond_lock (&(futex))
+#define lll_cond_lock(futex, private) __lll_cond_lock (&(futex), private)
-#define lll_robust_mutex_cond_lock(futex, id) \
- __lll_robust_mutex_lock (&(futex), (id) | FUTEX_WAITERS)
+#define lll_robust_cond_lock(futex, id, private) \
+ __lll_robust_lock (&(futex), (id) | FUTEX_WAITERS, private)
extern int __lll_timedlock_wait
- (int *futex, const struct timespec *) attribute_hidden;
+ (int *futex, const struct timespec *, int private) attribute_hidden;
extern int __lll_robust_timedlock_wait
- (int *futex, const struct timespec *) attribute_hidden;
+ (int *futex, const struct timespec *, int private) attribute_hidden;
static inline int
__attribute__ ((always_inline))
-__lll_mutex_timedlock (int *futex, const struct timespec *abstime)
+__lll_timedlock (int *futex, const struct timespec *abstime, int private)
{
int result = 0;
- if (atomic_compare_and_exchange_bool_acq (futex, 1, 0) != 0)
- result = __lll_timedlock_wait (futex, abstime);
+ if (__builtin_expect (atomic_compare_and_exchange_bool_acq (futex, 1, 0), 0))
+ result = __lll_timedlock_wait (futex, abstime, private);
return result;
}
-#define lll_mutex_timedlock(futex, abstime) \
- __lll_mutex_timedlock (&(futex), abstime)
+#define lll_timedlock(futex, abstime, private) \
+ __lll_timedlock (&(futex), abstime, private)
static inline int
__attribute__ ((always_inline))
-__lll_robust_mutex_timedlock (int *futex, const struct timespec *abstime,
- int id)
+__lll_robust_timedlock (int *futex, const struct timespec *abstime,
+ int id, int private)
{
int result = 0;
- if (atomic_compare_and_exchange_bool_acq (futex, id, 0) != 0)
- result = __lll_robust_timedlock_wait (futex, abstime);
+ if (__builtin_expect (atomic_compare_and_exchange_bool_acq (futex, id, 0),
+ 0))
+ result = __lll_robust_timedlock_wait (futex, abstime, private);
return result;
}
-#define lll_robust_mutex_timedlock(futex, abstime, id) \
- __lll_robust_mutex_timedlock (&(futex), abstime, id)
+#define lll_robust_timedlock(futex, abstime, id, private) \
+ __lll_robust_timedlock (&(futex), abstime, id, private)
-static inline void
-__attribute__ ((always_inline))
-__lll_mutex_unlock (int *futex)
-{
- int oldval;
- int newval = 0;
-
- lll_compare_and_swap (futex, oldval, newval, "slr %2,%2");
- if (oldval > 1)
- lll_futex_wake (futex, 1);
-}
-#define lll_mutex_unlock(futex) \
- __lll_mutex_unlock(&(futex))
-
-
-static inline void
-__attribute__ ((always_inline))
-__lll_robust_mutex_unlock (int *futex, int mask)
-{
- int oldval;
- int newval = 0;
-
- lll_compare_and_swap (futex, oldval, newval, "slr %2,%2");
- if (oldval & mask)
- lll_futex_wake (futex, 1);
-}
-#define lll_robust_mutex_unlock(futex) \
- __lll_robust_mutex_unlock(&(futex), FUTEX_WAITERS)
+#define __lll_unlock(futex, private) \
+ (void) \
+ ({ int __oldval; \
+ int __newval = 0; \
+ int *__futexp = (futex); \
+ \
+ lll_compare_and_swap (__futexp, __oldval, __newval, "slr %2,%2"); \
+ if (__builtin_expect (__oldval > 1, 0)) \
+ lll_futex_wake (__futexp, 1, private); \
+ })
+#define lll_unlock(futex, private) __lll_unlock(&(futex), private)
-static inline void
-__attribute__ ((always_inline))
-__lll_mutex_unlock_force (int *futex)
-{
- *futex = 0;
- lll_futex_wake (futex, 1);
-}
-#define lll_mutex_unlock_force(futex) \
- __lll_mutex_unlock_force(&(futex))
+#define __lll_robust_unlock(futex, private) \
+ (void) \
+ ({ int __oldval; \
+ int __newval = 0; \
+ int *__futexp = (futex); \
+ \
+ lll_compare_and_swap (__futexp, __oldval, __newval, "slr %2,%2"); \
+ if (__builtin_expect (__oldval & FUTEX_WAITERS, 0)) \
+ lll_futex_wake (__futexp, 1, private); \
+ })
+#define lll_robust_unlock(futex, private) \
+ __lll_robust_unlock(&(futex), private)
-#define lll_mutex_islocked(futex) \
+#define lll_islocked(futex) \
(futex != 0)
-/* We have a separate internal lock implementation which is not tied
- to binary compatibility. We can use the lll_mutex_*. */
-
-/* Type for lock object. */
-typedef int lll_lock_t;
-
/* Initializers for lock. */
#define LLL_LOCK_INITIALIZER (0)
#define LLL_LOCK_INITIALIZER_LOCKED (1)
-#define lll_trylock(futex) lll_mutex_trylock (futex)
-#define lll_lock(futex) lll_mutex_lock (futex)
-#define lll_unlock(futex) lll_mutex_unlock (futex)
-#define lll_islocked(futex) lll_mutex_islocked (futex)
-
-extern int lll_unlock_wake_cb (int *__futex) attribute_hidden;
-
-/* The states of a lock are:
- 1 - untaken
- 0 - taken by one user
- <0 - taken by more users */
-
-
/* The kernel notifies a process with uses CLONE_CLEARTID via futex
wakeup when the clone terminates. The memory location contains the
thread ID while the clone is running and is reset to zero
afterwards. */
-static inline void
-__attribute__ ((always_inline))
-__lll_wait_tid (int *ptid)
-{
- int tid;
-
- while ((tid = *ptid) != 0)
- lll_futex_wait (ptid, tid);
-}
+#define __lll_wait_tid(ptid) \
+ do \
+ { \
+ int __tid; \
+ \
+ while ((__tid = *ptid) != 0) \
+ lll_futex_wait (ptid, __tid, LLL_SHARED); \
+ } \
+ while (0)
#define lll_wait_tid(tid) __lll_wait_tid(&(tid))
extern int __lll_timedwait_tid (int *, const struct timespec *)
@@ -355,25 +351,4 @@ extern int __lll_timedwait_tid (int *, c
__res; \
})
-/* Conditional variable handling. */
-
-extern void __lll_cond_wait (pthread_cond_t *cond)
- attribute_hidden;
-extern int __lll_cond_timedwait (pthread_cond_t *cond,
- const struct timespec *abstime)
- attribute_hidden;
-extern void __lll_cond_wake (pthread_cond_t *cond)
- attribute_hidden;
-extern void __lll_cond_broadcast (pthread_cond_t *cond)
- attribute_hidden;
-
-#define lll_cond_wait(cond) \
- __lll_cond_wait (cond)
-#define lll_cond_timedwait(cond, abstime) \
- __lll_cond_timedwait (cond, abstime)
-#define lll_cond_wake(cond) \
- __lll_cond_wake (cond)
-#define lll_cond_broadcast(cond) \
- __lll_cond_broadcast (cond)
-
#endif /* lowlevellock.h */
--- libc/nptl/sysdeps/unix/sysv/linux/s390/pthread_once.c 5 Feb 2003 09:29:49 -0000 1.1
+++ libc/nptl/sysdeps/unix/sysv/linux/s390/pthread_once.c 28 Jul 2007 19:32:36 -0000 1.2
@@ -30,7 +30,7 @@ clear_once_control (void *arg)
pthread_once_t *once_control = (pthread_once_t *) arg;
*once_control = 0;
- lll_futex_wake (once_control, INT_MAX);
+ lll_futex_wake (once_control, INT_MAX, LLL_PRIVATE);
}
@@ -76,7 +76,7 @@ __pthread_once (once_control, init_routi
if (((oldval ^ newval) & -4) == 0)
{
/* Same generation, some other thread was faster. Wait. */
- lll_futex_wait (once_control, newval);
+ lll_futex_wait (once_control, newval, LLL_PRIVATE);
continue;
}
}
@@ -101,7 +101,7 @@ __pthread_once (once_control, init_routi
: "m" (*once_control) : "cc" );
/* Wake up all other threads. */
- lll_futex_wake (once_control, INT_MAX);
+ lll_futex_wake (once_control, INT_MAX, LLL_PRIVATE);
break;
}
--- libc/nptl/sysdeps/unix/sysv/linux/s390/bits/pthreadtypes.h 15 Feb 2006 17:14:33 -0000 1.18
+++ libc/nptl/sysdeps/unix/sysv/linux/s390/bits/pthreadtypes.h 12 Aug 2007 19:38:58 -0000 1.19
@@ -159,9 +159,9 @@ typedef union
unsigned int __nr_readers_queued;
unsigned int __nr_writers_queued;
int __writer;
- int __pad1;
+ int __shared;
+ unsigned long int __pad1;
unsigned long int __pad2;
- unsigned long int __pad3;
/* FLAGS must stay at this position in the structure to maintain
binary compatibility. */
unsigned int __flags;
@@ -175,9 +175,12 @@ typedef union
unsigned int __writer_wakeup;
unsigned int __nr_readers_queued;
unsigned int __nr_writers_queued;
+ unsigned char __pad1;
+ unsigned char __pad2;
+ unsigned char __shared;
/* FLAGS must stay at this position in the structure to maintain
binary compatibility. */
- unsigned int __flags;
+ unsigned char __flags;
int __writer;
} __data;
# endif
--- libc/nptl/sysdeps/unix/sysv/linux/x86_64/libc-lowlevellock.S 11 Mar 2003 22:01:44 -0000 1.2
+++ libc/nptl/sysdeps/unix/sysv/linux/x86_64/libc-lowlevellock.S 1 Aug 2007 04:38:57 -0000 1.5
@@ -17,14 +17,4 @@
Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
02111-1307 USA. */
-/* In libc.so we do not unconditionally use the lock prefix. Only if
- the application is using threads. */
-#ifndef UP
-# define LOCK \
- cmpl $0, __libc_multiple_threads(%rip); \
- je 0f; \
- lock; \
-0:
-#endif
-
#include "lowlevellock.S"
--- libc/nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S 10 Feb 2007 00:04:24 -0000 1.17
+++ libc/nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S 1 Aug 2007 04:37:37 -0000 1.21
@@ -19,30 +19,58 @@
#include <sysdep.h>
#include <pthread-errnos.h>
+#include <kernel-features.h>
+#include <lowlevellock.h>
.text
-#ifndef LOCK
-# ifdef UP
-# define LOCK
+#ifdef __ASSUME_PRIVATE_FUTEX
+# define LOAD_PRIVATE_FUTEX_WAIT(reg) \
+ movl $(FUTEX_WAIT | FUTEX_PRIVATE_FLAG), reg
+# define LOAD_PRIVATE_FUTEX_WAKE(reg) \
+ movl $(FUTEX_WAKE | FUTEX_PRIVATE_FLAG), reg
+# define LOAD_FUTEX_WAIT(reg) \
+ xorl $(FUTEX_WAIT | FUTEX_PRIVATE_FLAG), reg
+# define LOAD_FUTEX_WAKE(reg) \
+ xorl $(FUTEX_WAKE | FUTEX_PRIVATE_FLAG), reg
+#else
+# if FUTEX_WAIT == 0
+# define LOAD_PRIVATE_FUTEX_WAIT(reg) \
+ movl %fs:PRIVATE_FUTEX, reg
+# else
+# define LOAD_PRIVATE_FUTEX_WAIT(reg) \
+ movl %fs:PRIVATE_FUTEX, reg ; \
+ orl $FUTEX_WAIT, reg
+# endif
+# define LOAD_PRIVATE_FUTEX_WAKE(reg) \
+ movl %fs:PRIVATE_FUTEX, reg ; \
+ orl $FUTEX_WAKE, reg
+# if FUTEX_WAIT == 0
+# define LOAD_FUTEX_WAIT(reg) \
+ xorl $FUTEX_PRIVATE_FLAG, reg ; \
+ andl %fs:PRIVATE_FUTEX, reg
# else
-# define LOCK lock
+# define LOAD_FUTEX_WAIT(reg) \
+ xorl $FUTEX_PRIVATE_FLAG, reg ; \
+ andl %fs:PRIVATE_FUTEX, reg ; \
+ orl $FUTEX_WAIT, reg
# endif
+# define LOAD_FUTEX_WAKE(reg) \
+ xorl $FUTEX_PRIVATE_FLAG, reg ; \
+ andl %fs:PRIVATE_FUTEX, reg ; \
+ orl $FUTEX_WAKE, reg
#endif
-#define SYS_futex 202
-#define FUTEX_WAIT 0
-#define FUTEX_WAKE 1
/* For the calculation see asm/vsyscall.h. */
#define VSYSCALL_ADDR_vgettimeofday 0xffffffffff600000
- .globl __lll_mutex_lock_wait
- .type __lll_mutex_lock_wait,@function
- .hidden __lll_mutex_lock_wait
+ .globl __lll_lock_wait_private
+ .type __lll_lock_wait_private,@function
+ .hidden __lll_lock_wait_private
.align 16
-__lll_mutex_lock_wait:
+__lll_lock_wait_private:
cfi_startproc
pushq %r10
cfi_adjust_cfa_offset(8)
@@ -52,11 +80,7 @@ __lll_mutex_lock_wait:
cfi_offset(%rdx, -24)
xorq %r10, %r10 /* No timeout. */
movl $2, %edx
-#if FUTEX_WAIT == 0
- xorl %esi, %esi
-#else
- movl $FUTEX_WAIT, %esi
-#endif
+ LOAD_PRIVATE_FUTEX_WAIT (%esi)
cmpl %edx, %eax /* NB: %edx == 2 */
jne 2f
@@ -78,15 +102,52 @@ __lll_mutex_lock_wait:
cfi_restore(%r10)
retq
cfi_endproc
- .size __lll_mutex_lock_wait,.-__lll_mutex_lock_wait
-
+ .size __lll_lock_wait_private,.-__lll_lock_wait_private
#ifdef NOT_IN_libc
- .globl __lll_mutex_timedlock_wait
- .type __lll_mutex_timedlock_wait,@function
- .hidden __lll_mutex_timedlock_wait
+ .globl __lll_lock_wait
+ .type __lll_lock_wait,@function
+ .hidden __lll_lock_wait
+ .align 16
+__lll_lock_wait:
+ cfi_startproc
+ pushq %r10
+ cfi_adjust_cfa_offset(8)
+ pushq %rdx
+ cfi_adjust_cfa_offset(8)
+ cfi_offset(%r10, -16)
+ cfi_offset(%rdx, -24)
+ xorq %r10, %r10 /* No timeout. */
+ movl $2, %edx
+ LOAD_FUTEX_WAIT (%esi)
+
+ cmpl %edx, %eax /* NB: %edx == 2 */
+ jne 2f
+
+1: movl $SYS_futex, %eax
+ syscall
+
+2: movl %edx, %eax
+ xchgl %eax, (%rdi) /* NB: lock is implied */
+
+ testl %eax, %eax
+ jnz 1b
+
+ popq %rdx
+ cfi_adjust_cfa_offset(-8)
+ cfi_restore(%rdx)
+ popq %r10
+ cfi_adjust_cfa_offset(-8)
+ cfi_restore(%r10)
+ retq
+ cfi_endproc
+ .size __lll_lock_wait,.-__lll_lock_wait
+
+ .globl __lll_timedlock_wait
+ .type __lll_timedlock_wait,@function
+ .hidden __lll_timedlock_wait
.align 16
-__lll_mutex_timedlock_wait:
+__lll_timedlock_wait:
cfi_startproc
/* Check for a valid timeout value. */
cmpq $1000000000, 8(%rdx)
@@ -107,10 +168,12 @@ __lll_mutex_timedlock_wait:
cfi_offset(%r12, -32)
cfi_offset(%r13, -40)
cfi_offset(%r14, -48)
+ pushq %rsi
+ cfi_adjust_cfa_offset(8)
/* Stack frame for the timespec and timeval structs. */
- subq $16, %rsp
- cfi_adjust_cfa_offset(16)
+ subq $24, %rsp
+ cfi_adjust_cfa_offset(24)
movq %rdi, %r12
movq %rdx, %r13
@@ -151,11 +214,8 @@ __lll_mutex_timedlock_wait:
je 8f
movq %rsp, %r10
-#if FUTEX_WAIT == 0
- xorl %esi, %esi
-#else
- movl $FUTEX_WAIT, %esi
-#endif
+ movl 24(%rsp), %esi
+ LOAD_FUTEX_WAIT (%esi)
movq %r12, %rdi
movl $SYS_futex, %eax
syscall
@@ -167,8 +227,8 @@ __lll_mutex_timedlock_wait:
cmpxchgl %edx, (%r12)
jnz 7f
-6: addq $16, %rsp
- cfi_adjust_cfa_offset(-16)
+6: addq $32, %rsp
+ cfi_adjust_cfa_offset(-32)
popq %r14
cfi_adjust_cfa_offset(-8)
cfi_restore(%r14)
@@ -189,7 +249,7 @@ __lll_mutex_timedlock_wait:
3: movl $EINVAL, %eax
retq
- cfi_adjust_cfa_offset(56)
+ cfi_adjust_cfa_offset(72)
cfi_offset(%r8, -16)
cfi_offset(%r9, -24)
cfi_offset(%r12, -32)
@@ -209,35 +269,45 @@ __lll_mutex_timedlock_wait:
5: movl $ETIMEDOUT, %eax
jmp 6b
cfi_endproc
- .size __lll_mutex_timedlock_wait,.-__lll_mutex_timedlock_wait
+ .size __lll_timedlock_wait,.-__lll_timedlock_wait
#endif
-#ifdef NOT_IN_libc
- .globl lll_unlock_wake_cb
- .type lll_unlock_wake_cb,@function
- .hidden lll_unlock_wake_cb
+ .globl __lll_unlock_wake_private
+ .type __lll_unlock_wake_private,@function
+ .hidden __lll_unlock_wake_private
.align 16
-lll_unlock_wake_cb:
+__lll_unlock_wake_private:
+ cfi_startproc
pushq %rsi
+ cfi_adjust_cfa_offset(8)
pushq %rdx
+ cfi_adjust_cfa_offset(8)
+ cfi_offset(%rsi, -16)
+ cfi_offset(%rdx, -24)
- LOCK
- addl $1, (%rdi)
- jng 1f
+ movl $0, (%rdi)
+ LOAD_PRIVATE_FUTEX_WAKE (%esi)
+ movl $1, %edx /* Wake one thread. */
+ movl $SYS_futex, %eax
+ syscall
popq %rdx
+ cfi_adjust_cfa_offset(-8)
+ cfi_restore(%rdx)
popq %rsi
+ cfi_adjust_cfa_offset(-8)
+ cfi_restore(%rsi)
retq
- .size lll_unlock_wake_cb,.-lll_unlock_wake_cb
-#endif
-
+ cfi_endproc
+ .size __lll_unlock_wake_private,.-__lll_unlock_wake_private
- .globl __lll_mutex_unlock_wake
- .type __lll_mutex_unlock_wake,@function
- .hidden __lll_mutex_unlock_wake
+#ifdef NOT_IN_libc
+ .globl __lll_unlock_wake
+ .type __lll_unlock_wake,@function
+ .hidden __lll_unlock_wake
.align 16
-__lll_mutex_unlock_wake:
+__lll_unlock_wake:
cfi_startproc
pushq %rsi
cfi_adjust_cfa_offset(8)
@@ -247,7 +317,7 @@ __lll_mutex_unlock_wake:
cfi_offset(%rdx, -24)
movl $0, (%rdi)
- movl $FUTEX_WAKE, %esi
+ LOAD_FUTEX_WAKE (%esi)
movl $1, %edx /* Wake one thread. */
movl $SYS_futex, %eax
syscall
@@ -260,10 +330,8 @@ __lll_mutex_unlock_wake:
cfi_restore(%rsi)
retq
cfi_endproc
- .size __lll_mutex_unlock_wake,.-__lll_mutex_unlock_wake
+ .size __lll_unlock_wake,.-__lll_unlock_wake
-
-#ifdef NOT_IN_libc
.globl __lll_timedwait_tid
.type __lll_timedwait_tid,@function
.hidden __lll_timedwait_tid
@@ -305,6 +373,8 @@ __lll_timedwait_tid:
jz 4f
movq %rsp, %r10
+ /* XXX The kernel so far uses global futex for the wakeup at
+ all times. */
#if FUTEX_WAIT == 0
xorl %esi, %esi
#else
--- libc/nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.h 12 Jan 2007 05:11:42 -0000 1.30
+++ libc/nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.h 11 Aug 2007 18:49:14 -0000 1.37
@@ -1,4 +1,4 @@
-/* Copyright (C) 2002, 2003, 2004, 2006 Free Software Foundation, Inc.
+/* Copyright (C) 2002, 2003, 2004, 2006, 2007 Free Software Foundation, Inc.
This file is part of the GNU C Library.
Contributed by Ulrich Drepper <drepper@redhat.com>, 2002.
@@ -20,33 +20,83 @@
#ifndef _LOWLEVELLOCK_H
#define _LOWLEVELLOCK_H 1
-#include <time.h>
-#include <sys/param.h>
-#include <bits/pthreadtypes.h>
-
-#ifndef LOCK_INSTR
-# ifdef UP
-# define LOCK_INSTR /* nothing */
-# else
-# define LOCK_INSTR "lock;"
+#ifndef __ASSEMBLER__
+# include <time.h>
+# include <sys/param.h>
+# include <bits/pthreadtypes.h>
+# include <kernel-features.h>
+# include <tcb-offsets.h>
+
+# ifndef LOCK_INSTR
+# ifdef UP
+# define LOCK_INSTR /* nothing */
+# else
+# define LOCK_INSTR "lock;"
+# endif
+# endif
+#else
+# ifndef LOCK
+# ifdef UP
+# define LOCK
+# else
+# define LOCK lock
+# endif
# endif
#endif
#define SYS_futex 202
#define FUTEX_WAIT 0
#define FUTEX_WAKE 1
+#define FUTEX_CMP_REQUEUE 4
+#define FUTEX_WAKE_OP 5
#define FUTEX_LOCK_PI 6
#define FUTEX_UNLOCK_PI 7
#define FUTEX_TRYLOCK_PI 8
+#define FUTEX_PRIVATE_FLAG 128
+#define FUTEX_OP_CLEAR_WAKE_IF_GT_ONE ((4 << 24) | 1)
+
+/* Values for 'private' parameter of locking macros. Yes, the
+ definition seems to be backwards. But it is not. The bit will be
+ reversed before passing to the system call. */
+#define LLL_PRIVATE 0
+#define LLL_SHARED FUTEX_PRIVATE_FLAG
+
+#ifndef __ASSEMBLER__
+
+#if !defined NOT_IN_libc || defined IS_IN_rtld
+/* In libc.so or ld.so all futexes are private. */
+# ifdef __ASSUME_PRIVATE_FUTEX
+# define __lll_private_flag(fl, private) \
+ ((fl) | FUTEX_PRIVATE_FLAG)
+# else
+# define __lll_private_flag(fl, private) \
+ ((fl) | THREAD_GETMEM (THREAD_SELF, header.private_futex))
+# endif
+#else
+# ifdef __ASSUME_PRIVATE_FUTEX
+# define __lll_private_flag(fl, private) \
+ (((fl) | FUTEX_PRIVATE_FLAG) ^ (private))
+# else
+# define __lll_private_flag(fl, private) \
+ (__builtin_constant_p (private) \
+ ? ((private) == 0 \
+ ? ((fl) | THREAD_GETMEM (THREAD_SELF, header.private_futex)) \
+ : (fl)) \
+ : ({ unsigned int __fl = ((private) ^ FUTEX_PRIVATE_FLAG); \
+ asm ("andl %%fs:%P1, %0" : "+r" (__fl) \
+ : "i" (offsetof (struct pthread, header.private_futex))); \
+ __fl | (fl); }))
+# endif
+#endif
-/* Initializer for compatibility lock. */
-#define LLL_MUTEX_LOCK_INITIALIZER (0)
-#define LLL_MUTEX_LOCK_INITIALIZER_LOCKED (1)
-#define LLL_MUTEX_LOCK_INITIALIZER_WAITERS (2)
+/* Initializer for lock. */
+#define LLL_LOCK_INITIALIZER (0)
+#define LLL_LOCK_INITIALIZER_LOCKED (1)
+#define LLL_LOCK_INITIALIZER_WAITERS (2)
/* Delay in spinlock loop. */
-#define BUSY_WAIT_NOP asm ("rep; nop")
+#define BUSY_WAIT_NOP asm ("rep; nop")
#define LLL_STUB_UNWIND_INFO_START \
@@ -147,390 +197,358 @@ LLL_STUB_UNWIND_INFO_START \
LLL_STUB_UNWIND_INFO_END
-#define lll_futex_wait(futex, val) \
- ({ \
- int __status; \
- register __typeof (val) _val asm ("edx") = (val); \
- __asm __volatile ("xorq %%r10, %%r10\n\t" \
- "syscall" \
- : "=a" (__status) \
- : "0" (SYS_futex), "D" (futex), "S" (FUTEX_WAIT), \
- "d" (_val) \
- : "memory", "cc", "r10", "r11", "cx"); \
- __status; \
- })
+#define lll_futex_wait(futex, val, private) \
+ lll_futex_timed_wait(futex, val, NULL, private)
-#define lll_futex_timed_wait(futex, val, timeout) \
+#define lll_futex_timed_wait(futex, val, timeout, private) \
({ \
- register const struct timespec *__to __asm__ ("r10") = timeout; \
+ register const struct timespec *__to __asm ("r10") = timeout; \
int __status; \
- register __typeof (val) _val asm ("edx") = (val); \
+ register __typeof (val) _val __asm ("edx") = (val); \
__asm __volatile ("syscall" \
: "=a" (__status) \
- : "0" (SYS_futex), "D" (futex), "S" (FUTEX_WAIT), \
- "d" (_val), "r" (__to) \
+ : "0" (SYS_futex), "D" (futex), \
+ "S" (__lll_private_flag (FUTEX_WAIT, private)), \
+ "d" (_val), "r" (__to) \
: "memory", "cc", "r11", "cx"); \
__status; \
})
-#define lll_futex_wake(futex, nr) \
+#define lll_futex_wake(futex, nr, private) \
do { \
int __ignore; \
- register __typeof (nr) _nr asm ("edx") = (nr); \
+ register __typeof (nr) _nr __asm ("edx") = (nr); \
__asm __volatile ("syscall" \
: "=a" (__ignore) \
- : "0" (SYS_futex), "D" (futex), "S" (FUTEX_WAKE), \
+ : "0" (SYS_futex), "D" (futex), \
+ "S" (__lll_private_flag (FUTEX_WAKE, private)), \
"d" (_nr) \
: "memory", "cc", "r10", "r11", "cx"); \
} while (0)
-/* Does not preserve %eax and %ecx. */
-extern int __lll_mutex_lock_wait (int *__futex, int __val) attribute_hidden;
-/* Does not preserver %eax, %ecx, and %edx. */
-extern int __lll_mutex_timedlock_wait (int *__futex, int __val,
- const struct timespec *__abstime)
- attribute_hidden;
-/* Preserves all registers but %eax. */
-extern int __lll_mutex_unlock_wait (int *__futex) attribute_hidden;
-
-
-/* NB: in the lll_mutex_trylock macro we simply return the value in %eax
+/* NB: in the lll_trylock macro we simply return the value in %eax
after the cmpxchg instruction. In case the operation succeded this
value is zero. In case the operation failed, the cmpxchg instruction
has loaded the current value of the memory work which is guaranteed
to be nonzero. */
-#define lll_mutex_trylock(futex) \
+#if defined NOT_IN_libc || defined UP
+# define __lll_trylock_asm LOCK_INSTR "cmpxchgl %2, %1"
+#else
+# define __lll_trylock_asm "cmpl $0, __libc_multiple_threads(%%rip)\n\t" \
+ "je 0f\n\t" \
+ "lock; cmpxchgl %2, %1\n\t" \
+ "jmp 1f\n\t" \
+ "0:\tcmpxchgl %2, %1\n\t" \
+ "1:"
+#endif
+
+#define lll_trylock(futex) \
({ int ret; \
- __asm __volatile (LOCK_INSTR "cmpxchgl %2, %1" \
+ __asm __volatile (__lll_trylock_asm \
: "=a" (ret), "=m" (futex) \
- : "r" (LLL_MUTEX_LOCK_INITIALIZER_LOCKED), "m" (futex),\
- "0" (LLL_MUTEX_LOCK_INITIALIZER) \
+ : "r" (LLL_LOCK_INITIALIZER_LOCKED), "m" (futex), \
+ "0" (LLL_LOCK_INITIALIZER) \
: "memory"); \
ret; })
-
-#define lll_robust_mutex_trylock(futex, id) \
+#define lll_robust_trylock(futex, id) \
({ int ret; \
__asm __volatile (LOCK_INSTR "cmpxchgl %2, %1" \
: "=a" (ret), "=m" (futex) \
- : "r" (id), "m" (futex), \
- "0" (LLL_MUTEX_LOCK_INITIALIZER) \
+ : "r" (id), "m" (futex), "0" (LLL_LOCK_INITIALIZER) \
: "memory"); \
ret; })
-
-#define lll_mutex_cond_trylock(futex) \
+#define lll_cond_trylock(futex) \
({ int ret; \
__asm __volatile (LOCK_INSTR "cmpxchgl %2, %1" \
: "=a" (ret), "=m" (futex) \
- : "r" (LLL_MUTEX_LOCK_INITIALIZER_WAITERS), \
- "m" (futex), "0" (LLL_MUTEX_LOCK_INITIALIZER) \
+ : "r" (LLL_LOCK_INITIALIZER_WAITERS), \
+ "m" (futex), "0" (LLL_LOCK_INITIALIZER) \
: "memory"); \
ret; })
-
-#define lll_mutex_lock(futex) \
- (void) ({ int ignore1, ignore2, ignore3; \
- __asm __volatile (LOCK_INSTR "cmpxchgl %0, %2\n\t" \
+#if defined NOT_IN_libc || defined UP
+# define __lll_lock_asm_start LOCK_INSTR "cmpxchgl %4, %2\n\t" \
+ "jnz 1f\n\t"
+#else
+# define __lll_lock_asm_start "cmpl $0, __libc_multiple_threads(%%rip)\n\t" \
+ "je 0f\n\t" \
+ "lock; cmpxchgl %4, %2\n\t" \
"jnz 1f\n\t" \
- ".subsection 1\n\t" \
- ".type _L_mutex_lock_%=, @function\n" \
- "_L_mutex_lock_%=:\n" \
- "1:\tleaq %2, %%rdi\n" \
- "2:\tsubq $128, %%rsp\n" \
- "3:\tcallq __lll_mutex_lock_wait\n" \
- "4:\taddq $128, %%rsp\n" \
- "5:\tjmp 24f\n" \
- "6:\t.size _L_mutex_lock_%=, 6b-1b\n\t" \
- ".previous\n" \
- LLL_STUB_UNWIND_INFO_5 \
- "24:" \
- : "=S" (ignore1), "=&D" (ignore2), "=m" (futex),\
- "=a" (ignore3) \
- : "0" (1), "m" (futex), "3" (0) \
- : "cx", "r11", "cc", "memory"); })
+ "jmp 24f\n" \
+ "0:\tcmpxchgl %4, %2\n\t" \
+ "jnz 1f\n\t"
+#endif
+#define lll_lock(futex, private) \
+ (void) \
+ ({ int ignore1, ignore2, ignore3; \
+ if (__builtin_constant_p (private) && (private) == LLL_PRIVATE) \
+ __asm __volatile (__lll_lock_asm_start \
+ ".subsection 1\n\t" \
+ ".type _L_lock_%=, @function\n" \
+ "_L_lock_%=:\n" \
+ "1:\tleaq %2, %%rdi\n" \
+ "2:\tsubq $128, %%rsp\n" \
+ "3:\tcallq __lll_lock_wait_private\n" \
+ "4:\taddq $128, %%rsp\n" \
+ "5:\tjmp 24f\n" \
+ "6:\t.size _L_lock_%=, 6b-1b\n\t" \
+ ".previous\n" \
+ LLL_STUB_UNWIND_INFO_5 \
+ "24:" \
+ : "=S" (ignore1), "=&D" (ignore2), "=m" (futex), \
+ "=a" (ignore3) \
+ : "0" (1), "m" (futex), "3" (0) \
+ : "cx", "r11", "cc", "memory"); \
+ else \
+ __asm __volatile (__lll_lock_asm_start \
+ ".subsection 1\n\t" \
+ ".type _L_lock_%=, @function\n" \
+ "_L_lock_%=:\n" \
+ "1:\tleaq %2, %%rdi\n" \
+ "2:\tsubq $128, %%rsp\n" \
+ "3:\tcallq __lll_lock_wait\n" \
+ "4:\taddq $128, %%rsp\n" \
+ "5:\tjmp 24f\n" \
+ "6:\t.size _L_lock_%=, 6b-1b\n\t" \
+ ".previous\n" \
+ LLL_STUB_UNWIND_INFO_5 \
+ "24:" \
+ : "=S" (ignore1), "=D" (ignore2), "=m" (futex), \
+ "=a" (ignore3) \
+ : "1" (1), "m" (futex), "3" (0), "0" (private) \
+ : "cx", "r11", "cc", "memory"); \
+ }) \
-#define lll_robust_mutex_lock(futex, id) \
+#define lll_robust_lock(futex, id, private) \
({ int result, ignore1, ignore2; \
- __asm __volatile (LOCK_INSTR "cmpxchgl %0, %2\n\t" \
+ __asm __volatile (LOCK_INSTR "cmpxchgl %4, %2\n\t" \
"jnz 1f\n\t" \
".subsection 1\n\t" \
- ".type _L_robust_mutex_lock_%=, @function\n" \
- "_L_robust_mutex_lock_%=:\n" \
+ ".type _L_robust_lock_%=, @function\n" \
+ "_L_robust_lock_%=:\n" \
"1:\tleaq %2, %%rdi\n" \
"2:\tsubq $128, %%rsp\n" \
- "3:\tcallq __lll_robust_mutex_lock_wait\n" \
+ "3:\tcallq __lll_robust_lock_wait\n" \
"4:\taddq $128, %%rsp\n" \
"5:\tjmp 24f\n" \
- "6:\t.size _L_robust_mutex_lock_%=, 6b-1b\n\t" \
+ "6:\t.size _L_robust_lock_%=, 6b-1b\n\t" \
".previous\n" \
LLL_STUB_UNWIND_INFO_5 \
"24:" \
- : "=S" (ignore1), "=&D" (ignore2), "=m" (futex), \
+ : "=S" (ignore1), "=D" (ignore2), "=m" (futex), \
"=a" (result) \
- : "0" (id), "m" (futex), "3" (0) \
+ : "1" (id), "m" (futex), "3" (0), "0" (private) \
: "cx", "r11", "cc", "memory"); \
result; })
+#define lll_cond_lock(futex, private) \
+ (void) \
+ ({ int ignore1, ignore2, ignore3; \
+ __asm __volatile (LOCK_INSTR "cmpxchgl %4, %2\n\t" \
+ "jnz 1f\n\t" \
+ ".subsection 1\n\t" \
+ ".type _L_cond_lock_%=, @function\n" \
+ "_L_cond_lock_%=:\n" \
+ "1:\tleaq %2, %%rdi\n" \
+ "2:\tsubq $128, %%rsp\n" \
+ "3:\tcallq __lll_lock_wait\n" \
+ "4:\taddq $128, %%rsp\n" \
+ "5:\tjmp 24f\n" \
+ "6:\t.size _L_cond_lock_%=, 6b-1b\n\t" \
+ ".previous\n" \
+ LLL_STUB_UNWIND_INFO_5 \
+ "24:" \
+ : "=S" (ignore1), "=D" (ignore2), "=m" (futex), \
+ "=a" (ignore3) \
+ : "1" (2), "m" (futex), "3" (0), "0" (private) \
+ : "cx", "r11", "cc", "memory"); \
+ })
-#define lll_mutex_cond_lock(futex) \
- (void) ({ int ignore1, ignore2, ignore3; \
- __asm __volatile (LOCK_INSTR "cmpxchgl %0, %2\n\t" \
- "jnz 1f\n\t" \
- ".subsection 1\n\t" \
- ".type _L_mutex_cond_lock_%=, @function\n" \
- "_L_mutex_cond_lock_%=:\n" \
- "1:\tleaq %2, %%rdi\n" \
- "2:\tsubq $128, %%rsp\n" \
- "3:\tcallq __lll_mutex_lock_wait\n" \
- "4:\taddq $128, %%rsp\n" \
- "5:\tjmp 24f\n" \
- "6:\t.size _L_mutex_cond_lock_%=, 6b-1b\n\t" \
- ".previous\n" \
- LLL_STUB_UNWIND_INFO_5 \
- "24:" \
- : "=S" (ignore1), "=&D" (ignore2), "=m" (futex),\
- "=a" (ignore3) \
- : "0" (2), "m" (futex), "3" (0) \
- : "cx", "r11", "cc", "memory"); })
-
-
-#define lll_robust_mutex_cond_lock(futex, id) \
+#define lll_robust_cond_lock(futex, id, private) \
({ int result, ignore1, ignore2; \
- __asm __volatile (LOCK_INSTR "cmpxchgl %0, %2\n\t" \
+ __asm __volatile (LOCK_INSTR "cmpxchgl %4, %2\n\t" \
"jnz 1f\n\t" \
".subsection 1\n\t" \
- ".type _L_robust_mutex_cond_lock_%=, @function\n" \
- "_L_robust_mutex_cond_lock_%=:\n" \
+ ".type _L_robust_cond_lock_%=, @function\n" \
+ "_L_robust_cond_lock_%=:\n" \
"1:\tleaq %2, %%rdi\n" \
"2:\tsubq $128, %%rsp\n" \
- "3:\tcallq __lll_robust_mutex_lock_wait\n" \
+ "3:\tcallq __lll_robust_lock_wait\n" \
"4:\taddq $128, %%rsp\n" \
"5:\tjmp 24f\n" \
- "6:\t.size _L_robust_mutex_cond_lock_%=, 6b-1b\n\t" \
+ "6:\t.size _L_robust_cond_lock_%=, 6b-1b\n\t" \
".previous\n" \
LLL_STUB_UNWIND_INFO_5 \
"24:" \
- : "=S" (ignore1), "=&D" (ignore2), "=m" (futex), \
+ : "=S" (ignore1), "=D" (ignore2), "=m" (futex), \
"=a" (result) \
- : "0" (id | FUTEX_WAITERS), "m" (futex), "3" (0) \
+ : "1" (id | FUTEX_WAITERS), "m" (futex), "3" (0), \
+ "0" (private) \
: "cx", "r11", "cc", "memory"); \
result; })
-
-#define lll_mutex_timedlock(futex, timeout) \
+#define lll_timedlock(futex, timeout, private) \
({ int result, ignore1, ignore2, ignore3; \
- __asm __volatile (LOCK_INSTR "cmpxchgl %2, %4\n\t" \
+ __asm __volatile (LOCK_INSTR "cmpxchgl %1, %4\n\t" \
"jnz 1f\n\t" \
".subsection 1\n\t" \
- ".type _L_mutex_timedlock_%=, @function\n" \
- "_L_mutex_timedlock_%=:\n" \
+ ".type _L_timedlock_%=, @function\n" \
+ "_L_timedlock_%=:\n" \
"1:\tleaq %4, %%rdi\n" \
"0:\tmovq %8, %%rdx\n" \
"2:\tsubq $128, %%rsp\n" \
- "3:\tcallq __lll_mutex_timedlock_wait\n" \
+ "3:\tcallq __lll_timedlock_wait\n" \
"4:\taddq $128, %%rsp\n" \
"5:\tjmp 24f\n" \
- "6:\t.size _L_mutex_timedlock_%=, 6b-1b\n\t" \
+ "6:\t.size _L_timedlock_%=, 6b-1b\n\t" \
".previous\n" \
LLL_STUB_UNWIND_INFO_6 \
"24:" \
- : "=a" (result), "=&D" (ignore1), "=S" (ignore2), \
+ : "=a" (result), "=D" (ignore1), "=S" (ignore2), \
"=&d" (ignore3), "=m" (futex) \
- : "0" (0), "2" (1), "m" (futex), "m" (timeout) \
+ : "0" (0), "1" (1), "m" (futex), "m" (timeout), \
+ "2" (private) \
: "memory", "cx", "cc", "r10", "r11"); \
result; })
-
-#define lll_robust_mutex_timedlock(futex, timeout, id) \
+#define lll_robust_timedlock(futex, timeout, id, private) \
({ int result, ignore1, ignore2, ignore3; \
- __asm __volatile (LOCK_INSTR "cmpxchgl %2, %4\n\t" \
+ __asm __volatile (LOCK_INSTR "cmpxchgl %1, %4\n\t" \
"jnz 1f\n\t" \
".subsection 1\n\t" \
- ".type _L_robust_mutex_timedlock_%=, @function\n" \
- "_L_robust_mutex_timedlock_%=:\n" \
+ ".type _L_robust_timedlock_%=, @function\n" \
+ "_L_robust_timedlock_%=:\n" \
"1:\tleaq %4, %%rdi\n" \
"0:\tmovq %8, %%rdx\n" \
"2:\tsubq $128, %%rsp\n" \
- "3:\tcallq __lll_robust_mutex_timedlock_wait\n" \
+ "3:\tcallq __lll_robust_timedlock_wait\n" \
"4:\taddq $128, %%rsp\n" \
"5:\tjmp 24f\n" \
- "6:\t.size _L_robust_mutex_timedlock_%=, 6b-1b\n\t" \
+ "6:\t.size _L_robust_timedlock_%=, 6b-1b\n\t" \
".previous\n" \
LLL_STUB_UNWIND_INFO_6 \
"24:" \
- : "=a" (result), "=&D" (ignore1), "=S" (ignore2), \
+ : "=a" (result), "=D" (ignore1), "=S" (ignore2), \
"=&d" (ignore3), "=m" (futex) \
- : "0" (0), "2" (id), "m" (futex), "m" (timeout) \
+ : "0" (0), "1" (id), "m" (futex), "m" (timeout), \
+ "2" (private) \
: "memory", "cx", "cc", "r10", "r11"); \
result; })
-
-#define lll_mutex_unlock(futex) \
- (void) ({ int ignore; \
- __asm __volatile (LOCK_INSTR "decl %0\n\t" \
- "jne 1f\n\t" \
- ".subsection 1\n\t" \
- ".type _L_mutex_unlock_%=, @function\n" \
- "_L_mutex_unlock_%=:\n" \
- "1:\tleaq %0, %%rdi\n" \
- "2:\tsubq $128, %%rsp\n" \
- "3:\tcallq __lll_mutex_unlock_wake\n" \
- "4:\taddq $128, %%rsp\n" \
- "5:\tjmp 24f\n" \
- "6:\t.size _L_mutex_unlock_%=, 6b-1b\n\t" \
- ".previous\n" \
- LLL_STUB_UNWIND_INFO_5 \
- "24:" \
- : "=m" (futex), "=&D" (ignore) \
- : "m" (futex) \
- : "ax", "cx", "r11", "cc", "memory"); })
-
-
-#define lll_robust_mutex_unlock(futex) \
- (void) ({ int ignore; \
- __asm __volatile (LOCK_INSTR "andl %2, %0\n\t" \
- "jne 1f\n\t" \
- ".subsection 1\n\t" \
- ".type _L_robust_mutex_unlock_%=, @function\n" \
- "_L_robust_mutex_unlock_%=:\n" \
- "1:\tleaq %0, %%rdi\n" \
- "2:\tsubq $128, %%rsp\n" \
- "3:\tcallq __lll_mutex_unlock_wake\n" \
- "4:\taddq $128, %%rsp\n" \
- "5:\tjmp 24f\n" \
- "6:\t.size _L_robust_mutex_unlock_%=, 6b-1b\n\t"\
- ".previous\n" \
- LLL_STUB_UNWIND_INFO_5 \
- "24:" \
- : "=m" (futex), "=&D" (ignore) \
- : "i" (FUTEX_WAITERS), "m" (futex) \
- : "ax", "cx", "r11", "cc", "memory"); })
-
-
-#define lll_robust_mutex_dead(futex) \
- (void) ({ int ignore; \
- __asm __volatile (LOCK_INSTR "orl %3, (%2)\n\t" \
- "syscall" \
- : "=m" (futex), "=a" (ignore) \
- : "D" (&(futex)), "i" (FUTEX_OWNER_DIED), \
- "S" (FUTEX_WAKE), "1" (__NR_futex), \
- "d" (1) \
- : "cx", "r11", "cc", "memory"); })
-
-
-#define lll_mutex_islocked(futex) \
- (futex != LLL_MUTEX_LOCK_INITIALIZER)
-
-
-/* We have a separate internal lock implementation which is not tied
- to binary compatibility. */
-
-/* Type for lock object. */
-typedef int lll_lock_t;
-
-/* Initializers for lock. */
-#define LLL_LOCK_INITIALIZER (0)
-#define LLL_LOCK_INITIALIZER_LOCKED (1)
-
-
-extern int lll_unlock_wake_cb (int *__futex) attribute_hidden;
-
-
-/* The states of a lock are:
- 0 - untaken
- 1 - taken by one user
- 2 - taken by more users */
-
-
#if defined NOT_IN_libc || defined UP
-# define lll_trylock(futex) lll_mutex_trylock (futex)
-# define lll_lock(futex) lll_mutex_lock (futex)
-# define lll_unlock(futex) lll_mutex_unlock (futex)
+# define __lll_unlock_asm_start LOCK_INSTR "decl %0\n\t" \
+ "jne 1f\n\t"
#else
-/* Special versions of the macros for use in libc itself. They avoid
- the lock prefix when the thread library is not used.
-
- The code sequence to avoid unnecessary lock prefixes is what the AMD
- guys suggested. If you do not like it, bring it up with AMD.
-
- XXX In future we might even want to avoid it on UP machines. */
-
-# define lll_trylock(futex) \
- ({ unsigned char ret; \
- __asm __volatile ("cmpl $0, __libc_multiple_threads(%%rip)\n\t" \
- "je 0f\n\t" \
- "lock; cmpxchgl %2, %1\n\t" \
- "jmp 1f\n" \
- "0:\tcmpxchgl %2, %1\n\t" \
- "1:setne %0" \
- : "=a" (ret), "=m" (futex) \
- : "r" (LLL_MUTEX_LOCK_INITIALIZER_LOCKED), "m" (futex),\
- "0" (LLL_MUTEX_LOCK_INITIALIZER) \
- : "memory"); \
- ret; })
-
-
-# define lll_lock(futex) \
- (void) ({ int ignore1, ignore2, ignore3; \
- __asm __volatile ("cmpl $0, __libc_multiple_threads(%%rip)\n\t" \
- "je 0f\n\t" \
- "lock; cmpxchgl %0, %2\n\t" \
- "jnz 1f\n\t" \
- "jmp 24f\n" \
- "0:\tcmpxchgl %0, %2\n\t" \
- "jnz 1f\n\t" \
- ".subsection 1\n\t" \
- ".type _L_lock_%=, @function\n" \
- "_L_lock_%=:\n" \
- "1:\tleaq %2, %%rdi\n" \
- "2:\tsubq $128, %%rsp\n" \
- "3:\tcallq __lll_mutex_lock_wait\n" \
- "4:\taddq $128, %%rsp\n" \
- "5:\tjmp 24f\n" \
- "6:\t.size _L_lock_%=, 6b-1b\n\t" \
- ".previous\n" \
- LLL_STUB_UNWIND_INFO_5 \
- "24:" \
- : "=S" (ignore1), "=&D" (ignore2), "=m" (futex),\
- "=a" (ignore3) \
- : "0" (1), "m" (futex), "3" (0) \
- : "cx", "r11", "cc", "memory"); })
-
-
-# define lll_unlock(futex) \
- (void) ({ int ignore; \
- __asm __volatile ("cmpl $0, __libc_multiple_threads(%%rip)\n\t" \
- "je 0f\n\t" \
- "lock; decl %0\n\t" \
- "jne 1f\n\t" \
- "jmp 24f\n" \
- "0:\tdecl %0\n\t" \
- "jne 1f\n\t" \
- ".subsection 1\n\t" \
- ".type _L_unlock_%=, @function\n" \
- "_L_unlock_%=:\n" \
- "1:\tleaq %0, %%rdi\n" \
- "2:\tsubq $128, %%rsp\n" \
- "3:\tcallq __lll_mutex_unlock_wake\n" \
- "4:\taddq $128, %%rsp\n" \
- "5:\tjmp 24f\n" \
- "6:\t.size _L_unlock_%=, 6b-1b\n\t" \
- ".previous\n" \
- LLL_STUB_UNWIND_INFO_5 \
- "24:" \
- : "=m" (futex), "=&D" (ignore) \
- : "m" (futex) \
- : "ax", "cx", "r11", "cc", "memory"); })
+# define __lll_unlock_asm_start "cmpl $0, __libc_multiple_threads(%%rip)\n\t" \
+ "je 0f\n\t" \
+ "lock; decl %0\n\t" \
+ "jne 1f\n\t" \
+ "jmp 24f\n\t" \
+ "0:\tdecl %0\n\t" \
+ "jne 1f\n\t"
#endif
+#define lll_unlock(futex, private) \
+ (void) \
+ ({ int ignore; \
+ if (__builtin_constant_p (private) && (private) == LLL_PRIVATE) \
+ __asm __volatile (__lll_unlock_asm_start \
+ ".subsection 1\n\t" \
+ ".type _L_unlock_%=, @function\n" \
+ "_L_unlock_%=:\n" \
+ "1:\tleaq %0, %%rdi\n" \
+ "2:\tsubq $128, %%rsp\n" \
+ "3:\tcallq __lll_unlock_wake_private\n" \
+ "4:\taddq $128, %%rsp\n" \
+ "5:\tjmp 24f\n" \
+ "6:\t.size _L_unlock_%=, 6b-1b\n\t" \
+ ".previous\n" \
+ LLL_STUB_UNWIND_INFO_5 \
+ "24:" \
+ : "=m" (futex), "=&D" (ignore) \
+ : "m" (futex) \
+ : "ax", "cx", "r11", "cc", "memory"); \
+ else \
+ __asm __volatile (__lll_unlock_asm_start \
+ ".subsection 1\n\t" \
+ ".type _L_unlock_%=, @function\n" \
+ "_L_unlock_%=:\n" \
+ "1:\tleaq %0, %%rdi\n" \
+ "2:\tsubq $128, %%rsp\n" \
+ "3:\tcallq __lll_unlock_wake\n" \
+ "4:\taddq $128, %%rsp\n" \
+ "5:\tjmp 24f\n" \
+ "6:\t.size _L_unlock_%=, 6b-1b\n\t" \
+ ".previous\n" \
+ LLL_STUB_UNWIND_INFO_5 \
+ "24:" \
+ : "=m" (futex), "=&D" (ignore) \
+ : "m" (futex), "S" (private) \
+ : "ax", "cx", "r11", "cc", "memory"); \
+ })
+
+#define lll_robust_unlock(futex, private) \
+ do \
+ { \
+ int ignore; \
+ __asm __volatile (LOCK_INSTR "andl %2, %0\n\t" \
+ "jne 1f\n\t" \
+ ".subsection 1\n\t" \
+ ".type _L_robust_unlock_%=, @function\n" \
+ "_L_robust_unlock_%=:\n" \
+ "1:\tleaq %0, %%rdi\n" \
+ "2:\tsubq $128, %%rsp\n" \
+ "3:\tcallq __lll_unlock_wake\n" \
+ "4:\taddq $128, %%rsp\n" \
+ "5:\tjmp 24f\n" \
+ "6:\t.size _L_robust_unlock_%=, 6b-1b\n\t" \
+ ".previous\n" \
+ LLL_STUB_UNWIND_INFO_5 \
+ "24:" \
+ : "=m" (futex), "=&D" (ignore) \
+ : "i" (FUTEX_WAITERS), "m" (futex), \
+ "S" (private) \
+ : "ax", "cx", "r11", "cc", "memory"); \
+ } \
+ while (0)
+
+#define lll_robust_dead(futex, private) \
+ do \
+ { \
+ int ignore; \
+ __asm __volatile (LOCK_INSTR "orl %3, (%2)\n\t" \
+ "syscall" \
+ : "=m" (futex), "=a" (ignore) \
+ : "D" (&(futex)), "i" (FUTEX_OWNER_DIED), \
+ "S" (__lll_private_flag (FUTEX_WAKE, private)), \
+ "1" (__NR_futex), "d" (1) \
+ : "cx", "r11", "cc", "memory"); \
+ } \
+ while (0)
+
+/* Returns non-zero if error happened, zero if success. */
+#define lll_futex_requeue(ftx, nr_wake, nr_move, mutex, val, private) \
+ ({ int __res; \
+ register int __nr_move __asm ("r10") = nr_move; \
+ register void *__mutex __asm ("r8") = mutex; \
+ register int __val __asm ("r9") = val; \
+ __asm __volatile ("syscall" \
+ : "=a" (__res) \
+ : "0" (__NR_futex), "D" ((void *) ftx), \
+ "S" (__lll_private_flag (FUTEX_CMP_REQUEUE, \
+ private)), "d" (nr_wake), \
+ "r" (__nr_move), "r" (__mutex), "r" (__val) \
+ : "cx", "r11", "cc", "memory"); \
+ __res < 0; })
#define lll_islocked(futex) \
- (futex != LLL_MUTEX_LOCK_INITIALIZER)
+ (futex != LLL_LOCK_INITIALIZER)
/* The kernel notifies a process with uses CLONE_CLEARTID via futex
@@ -569,25 +587,6 @@ extern int __lll_timedwait_tid (int *tid
} \
__result; })
-
-/* Conditional variable handling. */
-
-extern void __lll_cond_wait (pthread_cond_t *cond) attribute_hidden;
-extern int __lll_cond_timedwait (pthread_cond_t *cond,
- const struct timespec *abstime)
- attribute_hidden;
-extern void __lll_cond_wake (pthread_cond_t *cond) attribute_hidden;
-extern void __lll_cond_broadcast (pthread_cond_t *cond) attribute_hidden;
-
-
-#define lll_cond_wait(cond) \
- __lll_cond_wait (cond)
-#define lll_cond_timedwait(cond, abstime) \
- __lll_cond_timedwait (cond, abstime)
-#define lll_cond_wake(cond) \
- __lll_cond_wake (cond)
-#define lll_cond_broadcast(cond) \
- __lll_cond_broadcast (cond)
-
+#endif /* !__ASSEMBLER__ */
#endif /* lowlevellock.h */
--- libc/nptl/sysdeps/unix/sysv/linux/x86_64/lowlevelrobustlock.S 5 Sep 2006 14:46:43 -0000 1.3
+++ libc/nptl/sysdeps/unix/sysv/linux/x86_64/lowlevelrobustlock.S 1 Aug 2007 04:36:44 -0000 1.4
@@ -19,33 +19,40 @@
#include <sysdep.h>
#include <pthread-errnos.h>
+#include <lowlevellock.h>
#include <lowlevelrobustlock.h>
+#include <kernel-features.h>
.text
-#ifndef LOCK
-# ifdef UP
-# define LOCK
+#define FUTEX_WAITERS 0x80000000
+#define FUTEX_OWNER_DIED 0x40000000
+
+#ifdef __ASSUME_PRIVATE_FUTEX
+# define LOAD_FUTEX_WAIT(reg) \
+ xorl $(FUTEX_WAIT | FUTEX_PRIVATE_FLAG), reg
+#else
+# if FUTEX_WAIT == 0
+# define LOAD_FUTEX_WAIT(reg) \
+ xorl $FUTEX_PRIVATE_FLAG, reg ; \
+ andl %fs:PRIVATE_FUTEX, reg
# else
-# define LOCK lock
+# define LOAD_FUTEX_WAIT(reg) \
+ xorl $FUTEX_PRIVATE_FLAG, reg ; \
+ andl %fs:PRIVATE_FUTEX, reg ; \
+ orl $FUTEX_WAIT, reg
# endif
#endif
-#define SYS_futex 202
-#define FUTEX_WAIT 0
-#define FUTEX_WAKE 1
-#define FUTEX_WAITERS 0x80000000
-#define FUTEX_OWNER_DIED 0x40000000
-
/* For the calculation see asm/vsyscall.h. */
#define VSYSCALL_ADDR_vgettimeofday 0xffffffffff600000
- .globl __lll_robust_mutex_lock_wait
- .type __lll_robust_mutex_lock_wait,@function
- .hidden __lll_robust_mutex_lock_wait
+ .globl __lll_robust_lock_wait
+ .type __lll_robust_lock_wait,@function
+ .hidden __lll_robust_lock_wait
.align 16
-__lll_robust_mutex_lock_wait:
+__lll_robust_lock_wait:
cfi_startproc
pushq %r10
cfi_adjust_cfa_offset(8)
@@ -55,11 +62,7 @@ __lll_robust_mutex_lock_wait:
cfi_offset(%rdx, -24)
xorq %r10, %r10 /* No timeout. */
-#if FUTEX_WAIT == 0
- xorl %esi, %esi
-#else
- movl $FUTEX_WAIT, %esi
-#endif
+ LOAD_FUTEX_WAIT (%esi)
4: movl %eax, %edx
orl $FUTEX_WAITERS, %edx
@@ -97,14 +100,14 @@ __lll_robust_mutex_lock_wait:
cfi_restore(%r10)
retq
cfi_endproc
- .size __lll_robust_mutex_lock_wait,.-__lll_robust_mutex_lock_wait
+ .size __lll_robust_lock_wait,.-__lll_robust_lock_wait
- .globl __lll_robust_mutex_timedlock_wait
- .type __lll_robust_mutex_timedlock_wait,@function
- .hidden __lll_robust_mutex_timedlock_wait
+ .globl __lll_robust_timedlock_wait
+ .type __lll_robust_timedlock_wait,@function
+ .hidden __lll_robust_timedlock_wait
.align 16
-__lll_robust_mutex_timedlock_wait:
+__lll_robust_timedlock_wait:
cfi_startproc
/* Check for a valid timeout value. */
cmpq $1000000000, 8(%rdx)
@@ -122,10 +125,12 @@ __lll_robust_mutex_timedlock_wait:
cfi_offset(%r9, -24)
cfi_offset(%r12, -32)
cfi_offset(%r13, -40)
+ pushq %rsi
+ cfi_adjust_cfa_offset(8)
/* Stack frame for the timespec and timeval structs. */
- subq $24, %rsp
- cfi_adjust_cfa_offset(24)
+ subq $32, %rsp
+ cfi_adjust_cfa_offset(32)
movq %rdi, %r12
movq %rdx, %r13
@@ -174,11 +179,8 @@ __lll_robust_mutex_timedlock_wait:
jnz 5f
2: movq %rsp, %r10
-#if FUTEX_WAIT == 0
- xorl %esi, %esi
-#else
- movl $FUTEX_WAIT, %esi
-#endif
+ movl 32(%rsp), %esi
+ LOAD_FUTEX_WAIT (%esi)
movq %r12, %rdi
movl $SYS_futex, %eax
syscall
@@ -195,8 +197,8 @@ __lll_robust_mutex_timedlock_wait:
cmpxchgl %edx, (%r12)
jnz 7f
-6: addq $24, %rsp
- cfi_adjust_cfa_offset(-24)
+6: addq $40, %rsp
+ cfi_adjust_cfa_offset(-40)
popq %r13
cfi_adjust_cfa_offset(-8)
cfi_restore(%r13)
@@ -214,7 +216,7 @@ __lll_robust_mutex_timedlock_wait:
3: movl $EINVAL, %eax
retq
- cfi_adjust_cfa_offset(56)
+ cfi_adjust_cfa_offset(72)
cfi_offset(%r8, -16)
cfi_offset(%r9, -24)
cfi_offset(%r12, -32)
@@ -226,4 +228,4 @@ __lll_robust_mutex_timedlock_wait:
8: movl $ETIMEDOUT, %eax
jmp 6b
cfi_endproc
- .size __lll_robust_mutex_timedlock_wait,.-__lll_robust_mutex_timedlock_wait
+ .size __lll_robust_timedlock_wait,.-__lll_robust_timedlock_wait
--- libc/nptl/sysdeps/unix/sysv/linux/x86_64/pthread_barrier_wait.S 31 Mar 2005 10:00:15 -0000 1.6
+++ libc/nptl/sysdeps/unix/sysv/linux/x86_64/pthread_barrier_wait.S 1 Aug 2007 04:40:11 -0000 1.8
@@ -18,18 +18,9 @@
02111-1307 USA. */
#include <sysdep.h>
+#include <lowlevellock.h>
#include <lowlevelbarrier.h>
-#define SYS_futex 202
-#define FUTEX_WAIT 0
-#define FUTEX_WAKE 1
-
-#ifndef UP
-# define LOCK lock
-#else
-# define LOCK
-#endif
-
.text
@@ -65,9 +56,10 @@ pthread_barrier_wait:
if the CURR_EVENT memory has meanwhile been changed. */
7:
#if FUTEX_WAIT == 0
- xorl %esi, %esi
+ movl PRIVATE(%rdi), %esi
#else
movl $FUTEX_WAIT, %esi
+ orl PRIVATE(%rdi), %esi
#endif
xorq %r10, %r10
8: movl $SYS_futex, %eax
@@ -116,6 +108,7 @@ pthread_barrier_wait:
so 0x7fffffff is the highest value. */
movl $0x7fffffff, %edx
movl $FUTEX_WAKE, %esi
+ orl PRIVATE(%rdi), %esi
movl $SYS_futex, %eax
syscall
@@ -140,21 +133,29 @@ pthread_barrier_wait:
retq
-1: addq $MUTEX, %rdi
- callq __lll_mutex_lock_wait
+1: movl PRIVATE(%rdi), %esi
+ addq $MUTEX, %rdi
+ xorl $LLL_SHARED, %esi
+ callq __lll_lock_wait
subq $MUTEX, %rdi
jmp 2b
-4: addq $MUTEX, %rdi
- callq __lll_mutex_unlock_wake
+4: movl PRIVATE(%rdi), %esi
+ addq $MUTEX, %rdi
+ xorl $LLL_SHARED, %esi
+ callq __lll_unlock_wake
jmp 5b
-6: addq $MUTEX, %rdi
- callq __lll_mutex_unlock_wake
+6: movl PRIVATE(%rdi), %esi
+ addq $MUTEX, %rdi
+ xorl $LLL_SHARED, %esi
+ callq __lll_unlock_wake
subq $MUTEX, %rdi
jmp 7b
-9: addq $MUTEX, %rdi
- callq __lll_mutex_unlock_wake
+9: movl PRIVATE(%rdi), %esi
+ addq $MUTEX, %rdi
+ xorl $LLL_SHARED, %esi
+ callq __lll_unlock_wake
jmp 10b
.size pthread_barrier_wait,.-pthread_barrier_wait
--- libc/nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_broadcast.S 29 Jul 2006 04:31:49 -0000 1.10
+++ libc/nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_broadcast.S 14 Aug 2007 02:27:03 -0000 1.13
@@ -19,23 +19,11 @@
#include <sysdep.h>
#include <shlib-compat.h>
+#include <lowlevellock.h>
#include <lowlevelcond.h>
#include <kernel-features.h>
#include <pthread-pi-defines.h>
-
-#ifdef UP
-# define LOCK
-#else
-# define LOCK lock
-#endif
-
-#define SYS_futex 202
-#define FUTEX_WAIT 0
-#define FUTEX_WAKE 1
-#define FUTEX_REQUEUE 3
-#define FUTEX_CMP_REQUEUE 4
-
-#define EINVAL 22
+#include <pthread-errnos.h>
.text
@@ -82,11 +70,18 @@ __pthread_cond_broadcast:
je 9f
/* XXX: The kernel so far doesn't support requeue to PI futex. */
- testl $PI_BIT, MUTEX_KIND(%r8)
+ /* XXX: The kernel only supports FUTEX_CMP_REQUEUE to the same
+ type of futex (private resp. shared). */
+ testl $(PI_BIT | PS_BIT), MUTEX_KIND(%r8)
jne 9f
/* Wake up all threads. */
- movl $FUTEX_CMP_REQUEUE, %esi
+#ifdef __ASSUME_PRIVATE_FUTEX
+ movl $(FUTEX_CMP_REQUEUE|FUTEX_PRIVATE_FLAG), %esi
+#else
+ movl %fs:PRIVATE_FUTEX, %esi
+ orl $FUTEX_CMP_REQUEUE, %esi
+#endif
movl $SYS_futex, %eax
movl $1, %edx
movl $0x7fffffff, %r10d
@@ -115,7 +110,11 @@ __pthread_cond_broadcast:
#if cond_lock != 0
addq $cond_lock, %rdi
#endif
- callq __lll_mutex_lock_wait
+ cmpq $-1, dep_mutex-cond_lock(%rdi)
+ movl $LLL_PRIVATE, %eax
+ movl $LLL_SHARED, %esi
+ cmovne %eax, %esi
+ callq __lll_lock_wait
#if cond_lock != 0
subq $cond_lock, %rdi
#endif
@@ -123,18 +122,36 @@ __pthread_cond_broadcast:
/* Unlock in loop requires wakeup. */
5: addq $cond_lock-cond_futex, %rdi
- callq __lll_mutex_unlock_wake
+ cmpq $-1, dep_mutex-cond_lock(%rdi)
+ movl $LLL_PRIVATE, %eax
+ movl $LLL_SHARED, %esi
+ cmovne %eax, %esi
+ callq __lll_unlock_wake
jmp 6b
/* Unlock in loop requires wakeup. */
7: addq $cond_lock-cond_futex, %rdi
- callq __lll_mutex_unlock_wake
+ cmpq $-1, %r8
+ movl $LLL_PRIVATE, %eax
+ movl $LLL_SHARED, %esi
+ cmovne %eax, %esi
+ callq __lll_unlock_wake
subq $cond_lock-cond_futex, %rdi
jmp 8b
9: /* The futex requeue functionality is not available. */
+ cmpq $-1, %r8
movl $0x7fffffff, %edx
- movl $FUTEX_WAKE, %esi
+#ifdef __ASSUME_PRIVATE_FUTEX
+ movl $FUTEX_WAKE, %eax
+ movl $(FUTEX_WAKE|FUTEX_PRIVATE_FLAG), %esi
+ cmove %eax, %esi
+#else
+ movl $0, %eax
+ movl %fs:PRIVATE_FUTEX, %esi
+ cmove %eax, %esi
+ orl $FUTEX_WAKE, %esi
+#endif
movl $SYS_futex, %eax
syscall
jmp 10b
--- libc/nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_signal.S 8 Sep 2005 17:40:52 -0000 1.9
+++ libc/nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_signal.S 14 Aug 2007 02:27:29 -0000 1.12
@@ -19,23 +19,10 @@
#include <sysdep.h>
#include <shlib-compat.h>
+#include <lowlevellock.h>
#include <lowlevelcond.h>
#include <kernel-features.h>
-
-#ifdef UP
-# define LOCK
-#else
-# define LOCK lock
-#endif
-
-#define SYS_futex 202
-#define FUTEX_WAIT 0
-#define FUTEX_WAKE 1
-#define FUTEX_WAKE_OP 5
-
-#define FUTEX_OP_CLEAR_WAKE_IF_GT_ONE ((4 << 24) | 1)
-
-#define EINVAL 22
+#include <pthread-errnos.h>
.text
@@ -68,10 +55,20 @@ __pthread_cond_signal:
addl $1, (%rdi)
/* Wake up one thread. */
- movl $FUTEX_WAKE_OP, %esi
- movl $SYS_futex, %eax
+ cmpq $-1, dep_mutex(%r8)
movl $1, %edx
+#ifdef __ASSUME_PRIVATE_FUTEX
+ movl $FUTEX_WAKE_OP, %eax
+ movl $(FUTEX_WAKE_OP|FUTEX_PRIVATE_FLAG), %esi
+ cmove %eax, %esi
+#else
+ movl $0, %eax
+ movl %fs:PRIVATE_FUTEX, %esi
+ cmove %eax, %esi
+ orl $FUTEX_WAKE_OP, %esi
+#endif
movl $1, %r10d
+ movl $SYS_futex, %eax
#if cond_lock != 0
addq $cond_lock, %r8
#endif
@@ -88,7 +85,9 @@ __pthread_cond_signal:
xorl %eax, %eax
retq
-7: movl $FUTEX_WAKE, %esi
+7: /* %esi should be either FUTEX_WAKE_OP or
+ FUTEX_WAKE_OP|FUTEX_PRIVATE_FLAG from the previous syscall. */
+ xorl $(FUTEX_WAKE ^ FUTEX_WAKE_OP), %esi
movl $SYS_futex, %eax
/* %rdx should be 1 already from $FUTEX_WAKE_OP syscall.
movl $1, %edx */
@@ -111,7 +110,11 @@ __pthread_cond_signal:
#if cond_lock != 0
addq $cond_lock, %rdi
#endif
- callq __lll_mutex_lock_wait
+ cmpq $-1, dep_mutex-cond_lock(%rdi)
+ movl $LLL_PRIVATE, %eax
+ movl $LLL_SHARED, %esi
+ cmovne %eax, %esi
+ callq __lll_lock_wait
#if cond_lock != 0
subq $cond_lock, %rdi
#endif
@@ -120,7 +123,14 @@ __pthread_cond_signal:
/* Unlock in loop requires wakeup. */
5:
movq %r8, %rdi
- callq __lll_mutex_unlock_wake
+#if cond_lock != 0
+ addq $cond_lock, %rdi
+#endif
+ cmpq $-1, dep_mutex-cond_lock(%rdi)
+ movl $LLL_PRIVATE, %eax
+ movl $LLL_SHARED, %esi
+ cmovne %eax, %esi
+ callq __lll_unlock_wake
jmp 6b
.size __pthread_cond_signal, .-__pthread_cond_signal
versioned_symbol (libpthread, __pthread_cond_signal, pthread_cond_signal,
--- libc/nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S 31 Mar 2005 10:00:15 -0000 1.24
+++ libc/nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S 20 Aug 2007 22:36:19 -0000 1.29
@@ -19,18 +19,11 @@
#include <sysdep.h>
#include <shlib-compat.h>
+#include <lowlevellock.h>
#include <lowlevelcond.h>
#include <pthread-errnos.h>
-#ifdef UP
-# define LOCK
-#else
-# define LOCK lock
-#endif
-
-#define SYS_futex 202
-#define FUTEX_WAIT 0
-#define FUTEX_WAKE 1
+#include <kernel-features.h>
/* For the calculation see asm/vsyscall.h. */
#define VSYSCALL_ADDR_vgettimeofday 0xffffffffff600000
@@ -111,7 +104,7 @@ __pthread_cond_timedwait:
movq 8(%rsp), %rdi
incq total_seq(%rdi)
incl cond_futex(%rdi)
- addl $(1 << clock_bits), cond_nwaiters(%rdi)
+ addl $(1 << nwaiters_shift), cond_nwaiters(%rdi)
/* Install cancellation handler. */
#ifdef PIC
@@ -137,7 +130,7 @@ __pthread_cond_timedwait:
structure stores the number minus 1. */
movq 8(%rsp), %rdi
movl cond_nwaiters(%rdi), %edi
- andl $((1 << clock_bits) - 1), %edi
+ andl $((1 << nwaiters_shift) - 1), %edi
/* Only clocks 0 and 1 are allowed so far. Both are handled in the
kernel. */
leaq 24(%rsp), %rsi
@@ -195,12 +188,20 @@ __pthread_cond_timedwait:
movl %eax, (%rsp)
leaq 24(%rsp), %r10
-#if FUTEX_WAIT == 0
- xorl %esi, %esi
-#else
- movl $FUTEX_WAIT, %esi
-#endif
+ cmpq $-1, dep_mutex(%rdi)
movq %r12, %rdx
+#ifdef __ASSUME_PRIVATE_FUTEX
+ movl $FUTEX_WAIT, %eax
+ movl $(FUTEX_WAIT|FUTEX_PRIVATE_FLAG), %esi
+ cmove %eax, %esi
+#else
+ movl $0, %eax
+ movl %fs:PRIVATE_FUTEX, %esi
+ cmove %eax, %esi
+# if FUTEX_WAIT != 0
+ orl $FUTEX_WAIT, %esi
+# endif
+#endif
addq $cond_futex, %rdi
movl $SYS_futex, %eax
syscall
@@ -250,19 +251,29 @@ __pthread_cond_timedwait:
9: xorq %r14, %r14
14: incq woken_seq(%rdi)
-24: subl $(1 << clock_bits), cond_nwaiters(%rdi)
+24: subl $(1 << nwaiters_shift), cond_nwaiters(%rdi)
/* Wake up a thread which wants to destroy the condvar object. */
cmpq $0xffffffffffffffff, total_seq(%rdi)
jne 25f
movl cond_nwaiters(%rdi), %eax
- andl $~((1 << clock_bits) - 1), %eax
+ andl $~((1 << nwaiters_shift) - 1), %eax
jne 25f
addq $cond_nwaiters, %rdi
- movl $SYS_futex, %eax
- movl $FUTEX_WAKE, %esi
+ cmpq $-1, dep_mutex-cond_nwaiters(%rdi)
movl $1, %edx
+#ifdef __ASSUME_PRIVATE_FUTEX
+ movl $FUTEX_WAKE, %eax
+ movl $(FUTEX_WAKE|FUTEX_PRIVATE_FLAG), %esi
+ cmove %eax, %esi
+#else
+ movl $0, %eax
+ movl %fs:PRIVATE_FUTEX, %esi
+ cmove %eax, %esi
+ orl $FUTEX_WAKE, %esi
+#endif
+ movl $SYS_futex, %eax
syscall
subq $cond_nwaiters, %rdi
@@ -301,7 +312,11 @@ __pthread_cond_timedwait:
#if cond_lock != 0
addq $cond_lock, %rdi
#endif
- callq __lll_mutex_lock_wait
+ cmpq $-1, dep_mutex-cond_lock(%rdi)
+ movl $LLL_PRIVATE, %eax
+ movl $LLL_SHARED, %esi
+ cmovne %eax, %esi
+ callq __lll_lock_wait
jmp 2b
/* Unlock in loop requires wakeup. */
@@ -309,7 +324,11 @@ __pthread_cond_timedwait:
#if cond_lock != 0
addq $cond_lock, %rdi
#endif
- callq __lll_mutex_unlock_wake
+ cmpq $-1, dep_mutex-cond_lock(%rdi)
+ movl $LLL_PRIVATE, %eax
+ movl $LLL_SHARED, %esi
+ cmovne %eax, %esi
+ callq __lll_unlock_wake
jmp 4b
/* Locking in loop failed. */
@@ -317,7 +336,11 @@ __pthread_cond_timedwait:
#if cond_lock != 0
addq $cond_lock, %rdi
#endif
- callq __lll_mutex_lock_wait
+ cmpq $-1, dep_mutex-cond_lock(%rdi)
+ movl $LLL_PRIVATE, %eax
+ movl $LLL_SHARED, %esi
+ cmovne %eax, %esi
+ callq __lll_lock_wait
#if cond_lock != 0
subq $cond_lock, %rdi
#endif
@@ -328,7 +351,11 @@ __pthread_cond_timedwait:
#if cond_lock != 0
addq $cond_lock, %rdi
#endif
- callq __lll_mutex_unlock_wake
+ cmpq $-1, dep_mutex-cond_lock(%rdi)
+ movl $LLL_PRIVATE, %eax
+ movl $LLL_SHARED, %esi
+ cmovne %eax, %esi
+ callq __lll_unlock_wake
jmp 11b
/* The initial unlocking of the mutex failed. */
@@ -345,7 +372,11 @@ __pthread_cond_timedwait:
#if cond_lock != 0
addq $cond_lock, %rdi
#endif
- callq __lll_mutex_unlock_wake
+ cmpq $-1, dep_mutex-cond_lock(%rdi)
+ movl $LLL_PRIVATE, %eax
+ movl $LLL_SHARED, %esi
+ cmovne %eax, %esi
+ callq __lll_unlock_wake
17: movq (%rsp), %rax
jmp 18b
--- libc/nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S 9 Sep 2006 11:21:23 -0000 1.24
+++ libc/nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S 2 Sep 2007 16:52:35 -0000 1.29
@@ -19,18 +19,11 @@
#include <sysdep.h>
#include <shlib-compat.h>
+#include <lowlevellock.h>
#include <lowlevelcond.h>
#include <tcb-offsets.h>
-#ifdef UP
-# define LOCK
-#else
-# define LOCK lock
-#endif
-
-#define SYS_futex 202
-#define FUTEX_WAIT 0
-#define FUTEX_WAKE 1
+#include <kernel-features.h>
.text
@@ -58,7 +51,11 @@ __condvar_cleanup:
#if cond_lock != 0
addq $cond_lock, %rdi
#endif
- callq __lll_mutex_lock_wait
+ cmpq $-1, dep_mutex-cond_lock(%rdi)
+ movl $LLL_PRIVATE, %eax
+ movl $LLL_SHARED, %esi
+ cmovne %eax, %esi
+ callq __lll_lock_wait
#if cond_lock != 0
subq $cond_lock, %rdi
#endif
@@ -77,20 +74,30 @@ __condvar_cleanup:
incl cond_futex(%rdi)
6: incq woken_seq(%rdi)
-3: subl $(1 << clock_bits), cond_nwaiters(%rdi)
+3: subl $(1 << nwaiters_shift), cond_nwaiters(%rdi)
/* Wake up a thread which wants to destroy the condvar object. */
xorq %r12, %r12
cmpq $0xffffffffffffffff, total_seq(%rdi)
jne 4f
movl cond_nwaiters(%rdi), %eax
- andl $~((1 << clock_bits) - 1), %eax
+ andl $~((1 << nwaiters_shift) - 1), %eax
jne 4f
addq $cond_nwaiters, %rdi
- movl $SYS_futex, %eax
- movl $FUTEX_WAKE, %esi
+ cmpq $-1, dep_mutex-cond_nwaiters(%rdi)
movl $1, %edx
+#ifdef __ASSUME_PRIVATE_FUTEX
+ movl $FUTEX_WAKE, %eax
+ movl $(FUTEX_WAKE|FUTEX_PRIVATE_FLAG), %esi
+ cmove %eax, %esi
+#else
+ movl $0, %eax
+ movl %fs:PRIVATE_FUTEX, %esi
+ cmove %eax, %esi
+ orl $FUTEX_WAKE, %esi
+#endif
+ movl $SYS_futex, %eax
syscall
subq $cond_nwaiters, %rdi
movl $1, %r12d
@@ -105,14 +112,28 @@ __condvar_cleanup:
#if cond_lock != 0
addq $cond_lock, %rdi
#endif
- callq __lll_mutex_unlock_wake
+ cmpq $-1, dep_mutex-cond_lock(%rdi)
+ movl $LLL_PRIVATE, %eax
+ movl $LLL_SHARED, %esi
+ cmovne %eax, %esi
+ callq __lll_unlock_wake
/* Wake up all waiters to make sure no signal gets lost. */
2: testq %r12, %r12
jnz 5f
addq $cond_futex, %rdi
- movl $FUTEX_WAKE, %esi
+ cmpq $-1, dep_mutex-cond_futex(%rdi)
movl $0x7fffffff, %edx
+#ifdef __ASSUME_PRIVATE_FUTEX
+ movl $FUTEX_WAKE, %eax
+ movl $(FUTEX_WAKE|FUTEX_PRIVATE_FLAG), %esi
+ cmove %eax, %esi
+#else
+ movl $0, %eax
+ movl %fs:PRIVATE_FUTEX, %esi
+ cmove %eax, %esi
+ orl $FUTEX_WAKE, %esi
+#endif
movl $SYS_futex, %eax
syscall
@@ -185,7 +206,7 @@ __pthread_cond_wait:
movq 8(%rsp), %rdi
incq total_seq(%rdi)
incl cond_futex(%rdi)
- addl $(1 << clock_bits), cond_nwaiters(%rdi)
+ addl $(1 << nwaiters_shift), cond_nwaiters(%rdi)
/* Install cancellation handler. */
#ifdef PIC
@@ -221,12 +242,20 @@ __pthread_cond_wait:
xorq %r10, %r10
movq %r12, %rdx
addq $cond_futex-cond_lock, %rdi
- movl $SYS_futex, %eax
-#if FUTEX_WAIT == 0
- xorl %esi, %esi
-#else
- movl $FUTEX_WAIT, %esi
+ cmpq $-1, dep_mutex-cond_futex(%rdi)
+#ifdef __ASSUME_PRIVATE_FUTEX
+ movl $FUTEX_WAIT, %eax
+ movl $(FUTEX_WAIT|FUTEX_PRIVATE_FLAG), %esi
+ cmove %eax, %esi
+#else
+ movl $FUTEX_WAIT, %eax
+ movl %fs:PRIVATE_FUTEX, %esi
+ cmove %eax, %esi
+# if FUTEX_WAIT != 0
+ orl $FUTEX_WAIT, %esi
+# endif
#endif
+ movl $SYS_futex, %eax
syscall
movl (%rsp), %edi
@@ -262,19 +291,29 @@ __pthread_cond_wait:
incq woken_seq(%rdi)
/* Unlock */
-16: subl $(1 << clock_bits), cond_nwaiters(%rdi)
+16: subl $(1 << nwaiters_shift), cond_nwaiters(%rdi)
/* Wake up a thread which wants to destroy the condvar object. */
cmpq $0xffffffffffffffff, total_seq(%rdi)
jne 17f
movl cond_nwaiters(%rdi), %eax
- andl $~((1 << clock_bits) - 1), %eax
+ andl $~((1 << nwaiters_shift) - 1), %eax
jne 17f
addq $cond_nwaiters, %rdi
- movl $SYS_futex, %eax
- movl $FUTEX_WAKE, %esi
+ cmpq $-1, dep_mutex-cond_nwaiters(%rdi)
movl $1, %edx
+#ifdef __ASSUME_PRIVATE_FUTEX
+ movl $FUTEX_WAKE, %eax
+ movl $(FUTEX_WAKE|FUTEX_PRIVATE_FLAG), %esi
+ cmove %eax, %esi
+#else
+ movl $0, %eax
+ movl %fs:PRIVATE_FUTEX, %esi
+ cmove %eax, %esi
+ orl $FUTEX_WAKE, %esi
+#endif
+ movl $SYS_futex, %eax
syscall
subq $cond_nwaiters, %rdi
@@ -307,7 +346,11 @@ __pthread_cond_wait:
#if cond_lock != 0
addq $cond_lock, %rdi
#endif
- callq __lll_mutex_lock_wait
+ cmpq $-1, dep_mutex-cond_lock(%rdi)
+ movl $LLL_PRIVATE, %eax
+ movl $LLL_SHARED, %esi
+ cmovne %eax, %esi
+ callq __lll_lock_wait
jmp 2b
/* Unlock in loop requires wakeup. */
@@ -315,7 +358,11 @@ __pthread_cond_wait:
#if cond_lock != 0
addq $cond_lock, %rdi
#endif
- callq __lll_mutex_unlock_wake
+ cmpq $-1, dep_mutex-cond_lock(%rdi)
+ movl $LLL_PRIVATE, %eax
+ movl $LLL_SHARED, %esi
+ cmovne %eax, %esi
+ callq __lll_unlock_wake
jmp 4b
/* Locking in loop failed. */
@@ -323,7 +370,11 @@ __pthread_cond_wait:
#if cond_lock != 0
addq $cond_lock, %rdi
#endif
- callq __lll_mutex_lock_wait
+ cmpq $-1, dep_mutex-cond_lock(%rdi)
+ movl $LLL_PRIVATE, %eax
+ movl $LLL_SHARED, %esi
+ cmovne %eax, %esi
+ callq __lll_lock_wait
#if cond_lock != 0
subq $cond_lock, %rdi
#endif
@@ -334,7 +385,11 @@ __pthread_cond_wait:
#if cond_lock != 0
addq $cond_lock, %rdi
#endif
- callq __lll_mutex_unlock_wake
+ cmpq $-1, dep_mutex-cond_lock(%rdi)
+ movl $LLL_PRIVATE, %eax
+ movl $LLL_SHARED, %esi
+ cmovne %eax, %esi
+ callq __lll_unlock_wake
jmp 11b
/* The initial unlocking of the mutex failed. */
@@ -346,12 +401,16 @@ __pthread_cond_wait:
#else
decl cond_lock(%rdi)
#endif
- jne 13f
+ je 13f
#if cond_lock != 0
addq $cond_lock, %rdi
#endif
- callq __lll_mutex_unlock_wake
+ cmpq $-1, dep_mutex-cond_lock(%rdi)
+ movl $LLL_PRIVATE, %eax
+ movl $LLL_SHARED, %esi
+ cmovne %eax, %esi
+ callq __lll_unlock_wake
13: movq %r10, %rax
jmp 14b
--- libc/nptl/sysdeps/unix/sysv/linux/x86_64/pthread_once.S 14 Oct 2005 18:47:31 -0000 1.12
+++ libc/nptl/sysdeps/unix/sysv/linux/x86_64/pthread_once.S 1 Aug 2007 04:35:21 -0000 1.14
@@ -17,14 +17,10 @@
Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
02111-1307 USA. */
-#ifndef UP
-# define LOCK lock
-#else
-# define LOCK
-#endif
+#include <kernel-features.h>
+#include <tcb-offsets.h>
+#include <lowlevellock.h>
-#define SYS_futex 202
-#define FUTEX_WAKE 1
.comm __fork_generation, 4, 4
@@ -74,10 +70,15 @@ __pthread_once:
jnz 3f /* Different for generation -> run initializer. */
/* Somebody else got here first. Wait. */
-#if FUTEX_WAIT == 0
- xorl %esi, %esi
+#ifdef __ASSUME_PRIVATE_FUTEX
+ movl $FUTEX_WAIT|FUTEX_PRIVATE_FLAG, %esi
#else
+# if FUTEX_WAIT == 0
+ movl %fs:PRIVATE_FUTEX, %esi
+# else
movl $FUTEX_WAIT, %esi
+ orl %fs:PRIVATE_FUTEX, %esi
+# endif
#endif
movl $SYS_futex, %eax
syscall
@@ -106,7 +107,12 @@ __pthread_once:
/* Wake up all other threads. */
movl $0x7fffffff, %edx
+#ifdef __ASSUME_PRIVATE_FUTEX
+ movl $FUTEX_WAKE|FUTEX_PRIVATE_FLAG, %esi
+#else
movl $FUTEX_WAKE, %esi
+ orl %fs:PRIVATE_FUTEX, %esi
+#endif
movl $SYS_futex, %eax
syscall
@@ -133,7 +139,12 @@ clear_once_control:
movl $0, (%rdi)
movl $0x7fffffff, %edx
+#ifdef __ASSUME_PRIVATE_FUTEX
+ movl $FUTEX_WAKE|FUTEX_PRIVATE_FLAG, %esi
+#else
movl $FUTEX_WAKE, %esi
+ orl %fs:PRIVATE_FUTEX, %esi
+#endif
movl $SYS_futex, %eax
syscall
--- libc/nptl/sysdeps/unix/sysv/linux/x86_64/pthread_rwlock_rdlock.S 31 Mar 2005 10:00:15 -0000 1.8
+++ libc/nptl/sysdeps/unix/sysv/linux/x86_64/pthread_rwlock_rdlock.S 1 Aug 2007 04:37:06 -0000 1.11
@@ -18,19 +18,10 @@
02111-1307 USA. */
#include <sysdep.h>
+#include <lowlevellock.h>
#include <lowlevelrwlock.h>
#include <pthread-errnos.h>
-
-
-#define SYS_futex 202
-#define FUTEX_WAIT 0
-#define FUTEX_WAKE 1
-
-#ifndef UP
-# define LOCK lock
-#else
-# define LOCK
-#endif
+#include <kernel-features.h>
.text
@@ -73,12 +64,20 @@ __pthread_rwlock_rdlock:
#endif
jne 10f
-11: addq $READERS_WAKEUP, %rdi
-#if FUTEX_WAIT == 0
- xorl %esi, %esi
-#else
+11:
+#ifdef __ASSUME_PRIVATE_FUTEX
+ movl $FUTEX_PRIVATE_FLAG|FUTEX_WAIT, %esi
+ xorl PSHARED(%rdi), %esi
+#else
+# if FUTEX_WAIT == 0
+ movl PSHARED(%rdi), %esi
+# else
movl $FUTEX_WAIT, %esi
+ orl PSHARED(%rdi), %esi
+# endif
+ xorl %fs:PRIVATE_FUTEX, %esi
#endif
+ addq $READERS_WAKEUP, %rdi
movl $SYS_futex, %eax
syscall
@@ -113,11 +112,11 @@ __pthread_rwlock_rdlock:
movq %rdx, %rax
retq
-1:
+1: movl PSHARED(%rdi), %esi
#if MUTEX != 0
addq $MUTEX, %rdi
#endif
- callq __lll_mutex_lock_wait
+ callq __lll_lock_wait
#if MUTEX != 0
subq $MUTEX, %rdi
#endif
@@ -129,11 +128,11 @@ __pthread_rwlock_rdlock:
movl $EDEADLK, %edx
jmp 9b
-6:
+6: movl PSHARED(%rdi), %esi
#if MUTEX != 0
addq $MUTEX, %rdi
#endif
- callq __lll_mutex_unlock_wake
+ callq __lll_unlock_wake
#if MUTEX != 0
subq $MUTEX, %rdi
#endif
@@ -149,21 +148,21 @@ __pthread_rwlock_rdlock:
movl $EAGAIN, %edx
jmp 9b
-10:
+10: movl PSHARED(%rdi), %esi
#if MUTEX != 0
addq $MUTEX, %rdi
#endif
- callq __lll_mutex_unlock_wake
+ callq __lll_unlock_wake
#if MUTEX != 0
subq $MUTEX, %rdi
#endif
jmp 11b
-12:
+12: movl PSHARED(%rdi), %esi
#if MUTEX == 0
addq $MUTEX, %rdi
#endif
- callq __lll_mutex_lock_wait
+ callq __lll_lock_wait
#if MUTEX != 0
subq $MUTEX, %rdi
#endif
--- libc/nptl/sysdeps/unix/sysv/linux/x86_64/pthread_rwlock_timedrdlock.S 31 Mar 2005 10:00:15 -0000 1.10
+++ libc/nptl/sysdeps/unix/sysv/linux/x86_64/pthread_rwlock_timedrdlock.S 1 Aug 2007 04:35:45 -0000 1.14
@@ -18,25 +18,15 @@
02111-1307 USA. */
#include <sysdep.h>
+#include <lowlevellock.h>
#include <lowlevelrwlock.h>
#include <pthread-errnos.h>
+#include <kernel-features.h>
-#define SYS_futex 202
-#define FUTEX_WAIT 0
-#define FUTEX_WAKE 1
-
/* For the calculation see asm/vsyscall.h. */
#define VSYSCALL_ADDR_vgettimeofday 0xffffffffff600000
-
-#ifndef UP
-# define LOCK lock
-#else
-# define LOCK
-#endif
-
-
.text
.globl pthread_rwlock_timedrdlock
@@ -112,10 +102,17 @@ pthread_rwlock_timedrdlock:
movq %rcx, (%rsp) /* Store relative timeout. */
movq %rdi, 8(%rsp)
-#if FUTEX_WAIT == 0
- xorl %esi, %esi
-#else
+#ifdef __ASSUME_PRIVATE_FUTEX
+ movl $FUTEX_PRIVATE_FLAG|FUTEX_WAIT, %esi
+ xorl PSHARED(%r12), %esi
+#else
+# if FUTEX_WAIT == 0
+ movl PSHARED(%r12), %esi
+# else
movl $FUTEX_WAIT, %esi
+ orl PSHARED(%r12), %esi
+# endif
+ xorl %fs:PRIVATE_FUTEX, %esi
#endif
movq %rsp, %r10
movl %r14d, %edx
@@ -163,11 +160,11 @@ pthread_rwlock_timedrdlock:
popq %r12
retq
-1:
+1: movl PSHARED(%rdi), %esi
#if MUTEX != 0
addq $MUTEX, %rdi
#endif
- callq __lll_mutex_lock_wait
+ callq __lll_lock_wait
jmp 2b
14: cmpl %fs:TID, %eax
@@ -175,13 +172,13 @@ pthread_rwlock_timedrdlock:
movl $EDEADLK, %edx
jmp 9b
-6:
+6: movl PSHARED(%r12), %esi
#if MUTEX == 0
movq %r12, %rdi
#else
leal MUTEX(%r12), %rdi
#endif
- callq __lll_mutex_unlock_wake
+ callq __lll_unlock_wake
jmp 7b
/* Overflow. */
@@ -194,22 +191,22 @@ pthread_rwlock_timedrdlock:
movl $EAGAIN, %edx
jmp 9b
-10:
+10: movl PSHARED(%r12), %esi
#if MUTEX == 0
movq %r12, %rdi
#else
leaq MUTEX(%r12), %rdi
#endif
- callq __lll_mutex_unlock_wake
+ callq __lll_unlock_wake
jmp 11b
-12:
+12: movl PSHARED(%r12), %esi
#if MUTEX == 0
movq %r12, %rdi
#else
leaq MUTEX(%r12), %rdi
#endif
- callq __lll_mutex_lock_wait
+ callq __lll_lock_wait
jmp 13b
16: movq $-ETIMEDOUT, %rdx
--- libc/nptl/sysdeps/unix/sysv/linux/x86_64/pthread_rwlock_timedwrlock.S 31 Mar 2005 10:00:15 -0000 1.8
+++ libc/nptl/sysdeps/unix/sysv/linux/x86_64/pthread_rwlock_timedwrlock.S 1 Aug 2007 04:41:38 -0000 1.12
@@ -18,24 +18,15 @@
02111-1307 USA. */
#include <sysdep.h>
+#include <lowlevellock.h>
#include <lowlevelrwlock.h>
#include <pthread-errnos.h>
+#include <kernel-features.h>
-#define SYS_futex 202
-#define FUTEX_WAIT 0
-#define FUTEX_WAKE 1
-
/* For the calculation see asm/vsyscall.h. */
#define VSYSCALL_ADDR_vgettimeofday 0xffffffffff600000
-#ifndef UP
-# define LOCK lock
-#else
-# define LOCK
-#endif
-
-
.text
.globl pthread_rwlock_timedwrlock
@@ -108,10 +99,17 @@ pthread_rwlock_timedwrlock:
movq %rcx, (%rsp) /* Store relative timeout. */
movq %rdi, 8(%rsp)
-#if FUTEX_WAIT == 0
- xorl %esi, %esi
-#else
+#ifdef __ASSUME_PRIVATE_FUTEX
+ movl $FUTEX_PRIVATE_FLAG|FUTEX_WAIT, %esi
+ xorl PSHARED(%r12), %esi
+#else
+# if FUTEX_WAIT == 0
+ movl PSHARED(%r12), %esi
+# else
movl $FUTEX_WAIT, %esi
+ orl PSHARED(%r12), %esi
+# endif
+ xorl %fs:PRIVATE_FUTEX, %esi
#endif
movq %rsp, %r10
movl %r14d, %edx
@@ -159,11 +157,11 @@ pthread_rwlock_timedwrlock:
popq %r12
retq
-1:
+1: movl PSHARED(%rdi), %esi
#if MUTEX != 0
addq $MUTEX, %rdi
#endif
- callq __lll_mutex_lock_wait
+ callq __lll_lock_wait
jmp 2b
14: cmpl %fs:TID, %eax
@@ -171,13 +169,13 @@ pthread_rwlock_timedwrlock:
20: movl $EDEADLK, %edx
jmp 9b
-6:
+6: movl PSHARED(%r12), %esi
#if MUTEX == 0
movq %r12, %rdi
#else
leal MUTEX(%r12), %rdi
#endif
- callq __lll_mutex_unlock_wake
+ callq __lll_unlock_wake
jmp 7b
/* Overflow. */
@@ -185,22 +183,22 @@ pthread_rwlock_timedwrlock:
movl $EAGAIN, %edx
jmp 9b
-10:
+10: movl PSHARED(%r12), %esi
#if MUTEX == 0
movq %r12, %rdi
#else
leaq MUTEX(%r12), %rdi
#endif
- callq __lll_mutex_unlock_wake
+ callq __lll_unlock_wake
jmp 11b
-12:
+12: movl PSHARED(%r12), %esi
#if MUTEX == 0
movq %r12, %rdi
#else
leaq MUTEX(%r12), %rdi
#endif
- callq __lll_mutex_lock_wait
+ callq __lll_lock_wait
jmp 13b
16: movq $-ETIMEDOUT, %rdx
--- libc/nptl/sysdeps/unix/sysv/linux/x86_64/pthread_rwlock_unlock.S 31 Mar 2005 10:00:15 -0000 1.6
+++ libc/nptl/sysdeps/unix/sysv/linux/x86_64/pthread_rwlock_unlock.S 1 Aug 2007 04:38:37 -0000 1.10
@@ -18,18 +18,9 @@
02111-1307 USA. */
#include <sysdep.h>
+#include <lowlevellock.h>
#include <lowlevelrwlock.h>
-
-
-#define SYS_futex 202
-#define FUTEX_WAIT 0
-#define FUTEX_WAKE 1
-
-#ifndef UP
-# define LOCK lock
-#else
-# define LOCK
-#endif
+#include <kernel-features.h>
.text
@@ -56,9 +47,8 @@ __pthread_rwlock_unlock:
5: movl $0, WRITER(%rdi)
- movl $1, %esi
+ movl $1, %edx
leaq WRITERS_WAKEUP(%rdi), %r10
- movq %rsi, %rdx
cmpl $0, WRITERS_QUEUED(%rdi)
jne 0f
@@ -78,7 +68,16 @@ __pthread_rwlock_unlock:
#endif
jne 7f
-8: movl $SYS_futex, %eax
+8:
+#ifdef __ASSUME_PRIVATE_FUTEX
+ movl $FUTEX_PRIVATE_FLAG|FUTEX_WAKE, %esi
+ xorl PSHARED(%rdi), %esi
+#else
+ movl $FUTEX_WAKE, %esi
+ orl PSHARED(%rdi), %esi
+ xorl %fs:PRIVATE_FUTEX, %esi
+#endif
+ movl $SYS_futex, %eax
movq %r10, %rdi
syscall
@@ -97,28 +96,28 @@ __pthread_rwlock_unlock:
4: xorl %eax, %eax
retq
-1:
+1: movl PSHARED(%rdi), %esi
#if MUTEX != 0
addq $MUTEX, %rdi
#endif
- callq __lll_mutex_lock_wait
+ callq __lll_lock_wait
#if MUTEX != 0
subq $MUTEX, %rdi
#endif
jmp 2b
-3:
+3: movl PSHARED(%rdi), %esi
#if MUTEX != 0
addq $MUTEX, %rdi
#endif
- callq __lll_mutex_unlock_wake
+ callq __lll_unlock_wake
jmp 4b
-7:
+7: movl PSHARED(%rdi), %esi
#if MUTEX != 0
addq $MUTEX, %rdi
#endif
- callq __lll_mutex_unlock_wake
+ callq __lll_unlock_wake
jmp 8b
.size __pthread_rwlock_unlock,.-__pthread_rwlock_unlock
--- libc/nptl/sysdeps/unix/sysv/linux/x86_64/pthread_rwlock_wrlock.S 31 Mar 2005 10:00:15 -0000 1.8
+++ libc/nptl/sysdeps/unix/sysv/linux/x86_64/pthread_rwlock_wrlock.S 1 Aug 2007 04:35:56 -0000 1.11
@@ -18,19 +18,10 @@
02111-1307 USA. */
#include <sysdep.h>
+#include <lowlevellock.h>
#include <lowlevelrwlock.h>
#include <pthread-errnos.h>
-
-
-#define SYS_futex 202
-#define FUTEX_WAIT 0
-#define FUTEX_WAKE 1
-
-#ifndef UP
-# define LOCK lock
-#else
-# define LOCK
-#endif
+#include <kernel-features.h>
.text
@@ -71,12 +62,20 @@ __pthread_rwlock_wrlock:
#endif
jne 10f
-11: addq $WRITERS_WAKEUP, %rdi
-#if FUTEX_WAIT == 0
- xorl %esi, %esi
-#else
+11:
+#ifdef __ASSUME_PRIVATE_FUTEX
+ movl $FUTEX_PRIVATE_FLAG|FUTEX_WAIT, %esi
+ xorl PSHARED(%rdi), %esi
+#else
+# if FUTEX_WAIT == 0
+ movl PSHARED(%rdi), %esi
+# else
movl $FUTEX_WAIT, %esi
+ orl PSHARED(%rdi), %esi
+# endif
+ xorl %fs:PRIVATE_FUTEX, %esi
#endif
+ addq $WRITERS_WAKEUP, %rdi
movl $SYS_futex, %eax
syscall
@@ -111,11 +110,11 @@ __pthread_rwlock_wrlock:
movq %rdx, %rax
retq
-1:
+1: movl PSHARED(%rdi), %esi
#if MUTEX != 0
addq $MUTEX, %rdi
#endif
- callq __lll_mutex_lock_wait
+ callq __lll_lock_wait
#if MUTEX != 0
subq $MUTEX, %rdi
#endif
@@ -126,32 +125,32 @@ __pthread_rwlock_wrlock:
movl $EDEADLK, %edx
jmp 9b
-6:
+6: movl PSHARED(%rdi), %esi
#if MUTEX != 0
addq $MUTEX, %rdi
#endif
- callq __lll_mutex_unlock_wake
+ callq __lll_unlock_wake
jmp 7b
4: decl WRITERS_QUEUED(%rdi)
movl $EAGAIN, %edx
jmp 9b
-10:
+10: movl PSHARED(%rdi), %esi
#if MUTEX != 0
addq $MUTEX, %rdi
#endif
- callq __lll_mutex_unlock_wake
+ callq __lll_unlock_wake
#if MUTEX != 0
subq $MUTEX, %rdi
#endif
jmp 11b
-12:
+12: movl PSHARED(%rdi), %esi
#if MUTEX != 0
addq $MUTEX, %rdi
#endif
- callq __lll_mutex_lock_wait
+ callq __lll_lock_wait
#if MUTEX != 0
subq $MUTEX, %rdi
#endif
--- libc/nptl/sysdeps/unix/sysv/linux/x86_64/sem_post.S 15 May 2007 06:31:57 -0000 1.8
+++ libc/nptl/sysdeps/unix/sysv/linux/x86_64/sem_post.S 1 Aug 2007 04:38:15 -0000 1.11
@@ -18,17 +18,10 @@
02111-1307 USA. */
#include <sysdep.h>
+#include <lowlevellock.h>
#include <shlib-compat.h>
#include <pthread-errnos.h>
-
-#ifndef UP
-# define LOCK lock
-#else
-# define
-#endif
-
-#define SYS_futex 202
-#define FUTEX_WAKE 1
+#include <structsem.h>
.text
@@ -37,19 +30,26 @@
.type sem_post,@function
.align 16
sem_post:
- movl $1, %edx
LOCK
- xaddl %edx, (%rdi)
+#if VALUE == 0
+ addl $1, (%rdi)
+#else
+ addl $1, VALUE(%rdi)
+#endif
+
+ cmpq $0, NWAITERS(%rdi)
+ je 2f
movl $SYS_futex, %eax
movl $FUTEX_WAKE, %esi
- incl %edx
+ orl PRIVATE(%rdi), %esi
+ movl $1, %edx
syscall
testq %rax, %rax
js 1f
- xorl %eax, %eax
+2: xorl %eax, %eax
retq
1:
--- libc/nptl/sysdeps/unix/sysv/linux/x86_64/sem_timedwait.S 15 May 2007 06:24:23 -0000 1.11
+++ libc/nptl/sysdeps/unix/sysv/linux/x86_64/sem_timedwait.S 1 Aug 2007 04:39:40 -0000 1.15
@@ -1,4 +1,4 @@
-/* Copyright (C) 2002, 2003, 2005 Free Software Foundation, Inc.
+/* Copyright (C) 2002, 2003, 2005, 2007 Free Software Foundation, Inc.
This file is part of the GNU C Library.
Contributed by Ulrich Drepper <drepper@redhat.com>, 2002.
@@ -18,41 +18,37 @@
02111-1307 USA. */
#include <sysdep.h>
+#include <lowlevellock.h>
#include <shlib-compat.h>
#include <pthread-errnos.h>
+#include <structsem.h>
-#ifndef UP
-# define LOCK lock
-#else
-# define
-#endif
-
-#define SYS_futex 202
/* For the calculation see asm/vsyscall.h. */
#define VSYSCALL_ADDR_vgettimeofday 0xffffffffff600000
-
.text
.globl sem_timedwait
.type sem_timedwait,@function
.align 16
- cfi_startproc
sem_timedwait:
- /* First check for cancellation. */
- movl %fs:CANCELHANDLING, %eax
- andl $0xfffffff9, %eax
- cmpl $8, %eax
- je 11f
-
+.LSTARTCODE:
+#if VALUE == 0
movl (%rdi), %eax
+#else
+ movl VALUE(%rdi), %eax
+#endif
2: testl %eax, %eax
je 1f
leaq -1(%rax), %rdx
LOCK
+#if VALUE == 0
cmpxchgl %edx, (%rdi)
+#else
+ cmpxchgl %edx, VALUE(%rdi)
+#endif
jne 2b
xorl %eax, %eax
@@ -60,25 +56,25 @@ sem_timedwait:
/* Check whether the timeout value is valid. */
1: pushq %r12
- cfi_adjust_cfa_offset(8)
+.Lpush_r12:
pushq %r13
- cfi_adjust_cfa_offset(8)
+.Lpush_r13:
pushq %r14
- cfi_adjust_cfa_offset(8)
+.Lpush_r14:
subq $24, %rsp
- cfi_adjust_cfa_offset(24)
+.Lsubq:
movq %rdi, %r12
- cfi_offset(12, -16) /* %r12 */
movq %rsi, %r13
- cfi_offset(13, -24) /* %r13 */
/* Check for invalid nanosecond field. */
cmpq $1000000000, 8(%r13)
movl $EINVAL, %r14d
- cfi_offset(14, -24) /* %r14 */
jae 6f
+ LOCK
+ addq $1, NWAITERS(%r12)
+
7: xorl %esi, %esi
movq %rsp, %rdi
movq $VSYSCALL_ADDR_vgettimeofday, %rax
@@ -102,12 +98,22 @@ sem_timedwait:
movq %rdi, (%rsp) /* Store relative timeout. */
movq %rsi, 8(%rsp)
+.LcleanupSTART:
call __pthread_enable_asynccancel
movl %eax, 16(%rsp)
movq %rsp, %r10
+#if VALUE == 0
movq %r12, %rdi
- xorl %esi, %esi
+#else
+ leaq VALUE(%r12), %rdi
+#endif
+#if FUTEX_WAIT == 0
+ movl PRIVATE(%rdi), %esi
+#else
+ movl $FUTEX_WAIT, %esi
+ orl PRIVATE(%rdi), %esi
+#endif
movl $SYS_futex, %eax
xorl %edx, %edx
syscall
@@ -115,39 +121,47 @@ sem_timedwait:
movl 16(%rsp), %edi
call __pthread_disable_asynccancel
+.LcleanupEND:
testq %r14, %r14
je 9f
cmpq $-EWOULDBLOCK, %r14
jne 3f
-9: movl (%r12), %eax
+9:
+#if VALUE == 0
+ movl (%r12), %eax
+#else
+ movl VALUE(%r12), %eax
+#endif
8: testl %eax, %eax
je 7b
leaq -1(%rax), %rcx
LOCK
+#if VALUE == 0
cmpxchgl %ecx, (%r12)
+#else
+ cmpxchgl %ecx, VALUE(%r12)
+#endif
jne 8b
xorl %eax, %eax
-10: addq $24, %rsp
- cfi_adjust_cfa_offset(-24)
+
+10: LOCK
+ subq $1, NWAITERS(%r12)
+
+ addq $24, %rsp
+.Laddq:
popq %r14
- cfi_adjust_cfa_offset(-8)
- cfi_restore(14)
+.Lpop_r14:
popq %r13
- cfi_adjust_cfa_offset(-8)
- cfi_restore(13)
+.Lpop_r13:
popq %r12
- cfi_adjust_cfa_offset(-8)
- cfi_restore(12)
+.Lpop_r12:
retq
- cfi_adjust_cfa_offset(48)
- cfi_offset(12, -16) /* %r12 */
- cfi_offset(13, -24) /* %r13 */
- cfi_offset(14, -32) /* %r14 */
+.Lafter_retq:
3: negq %r14
6:
#if USE___THREAD
@@ -160,16 +174,159 @@ sem_timedwait:
orl $-1, %eax
jmp 10b
- cfi_adjust_cfa_offset(-48)
- cfi_restore(14)
- cfi_restore(13)
- cfi_restore(12)
+ .size sem_timedwait,.-sem_timedwait
-11: /* Canceled. */
- movq $0xffffffffffffffff, %fs:RESULT
+
+ .type sem_timedwait_cleanup,@function
+sem_timedwait_cleanup:
LOCK
- orl $0x10, %fs:CANCELHANDLING
- movq %fs:CLEANUP_JMP_BUF, %rdi
- jmp HIDDEN_JUMPTARGET (__pthread_unwind)
- cfi_endproc
- .size sem_timedwait,.-sem_timedwait
+ subq $1, NWAITERS(%r12)
+ movq %rax, %rdi
+.LcallUR:
+ call _Unwind_Resume@PLT
+ hlt
+.LENDCODE:
+ .size sem_timedwait_cleanup,.-sem_timedwait_cleanup
+
+
+ .section .gcc_except_table,"a",@progbits
+.LexceptSTART:
+ .byte 0xff # @LPStart format (omit)
+ .byte 0xff # @TType format (omit)
+ .byte 0x01 # call-site format
+ # DW_EH_PE_uleb128
+ .uleb128 .Lcstend-.Lcstbegin
+.Lcstbegin:
+ .uleb128 .LcleanupSTART-.LSTARTCODE
+ .uleb128 .LcleanupEND-.LcleanupSTART
+ .uleb128 sem_timedwait_cleanup-.LSTARTCODE
+ .uleb128 0
+ .uleb128 .LcallUR-.LSTARTCODE
+ .uleb128 .LENDCODE-.LcallUR
+ .uleb128 0
+ .uleb128 0
+.Lcstend:
+
+
+ .section .eh_frame,"a",@progbits
+.LSTARTFRAME:
+ .long .LENDCIE-.LSTARTCIE # Length of the CIE.
+.LSTARTCIE:
+ .long 0 # CIE ID.
+ .byte 1 # Version number.
+#ifdef SHARED
+ .string "zPLR" # NUL-terminated augmentation
+ # string.
+#else
+ .string "zPL" # NUL-terminated augmentation
+ # string.
+#endif
+ .uleb128 1 # Code alignment factor.
+ .sleb128 -8 # Data alignment factor.
+ .byte 16 # Return address register
+ # column.
+#ifdef SHARED
+ .uleb128 7 # Augmentation value length.
+ .byte 0x9b # Personality: DW_EH_PE_pcrel
+ # + DW_EH_PE_sdata4
+ # + DW_EH_PE_indirect
+ .long DW.ref.__gcc_personality_v0-.
+ .byte 0x1b # LSDA Encoding: DW_EH_PE_pcrel
+ # + DW_EH_PE_sdata4.
+ .byte 0x1b # FDE Encoding: DW_EH_PE_pcrel
+ # + DW_EH_PE_sdata4.
+#else
+ .uleb128 10 # Augmentation value length.
+ .byte 0x0 # Personality: absolute
+ .quad __gcc_personality_v0
+ .byte 0x0 # LSDA Encoding: absolute
+#endif
+ .byte 0x0c # DW_CFA_def_cfa
+ .uleb128 7
+ .uleb128 8
+ .byte 0x90 # DW_CFA_offset, column 0x10
+ .uleb128 1
+ .align 8
+.LENDCIE:
+
+ .long .LENDFDE-.LSTARTFDE # Length of the FDE.
+.LSTARTFDE:
+ .long .LSTARTFDE-.LSTARTFRAME # CIE pointer.
+#ifdef SHARED
+ .long .LSTARTCODE-. # PC-relative start address
+ # of the code.
+ .long .LENDCODE-.LSTARTCODE # Length of the code.
+ .uleb128 4 # Augmentation size
+ .long .LexceptSTART-.
+#else
+ .quad .LSTARTCODE # Start address of the code.
+ .quad .LENDCODE-.LSTARTCODE # Length of the code.
+ .uleb128 8 # Augmentation size
+ .quad .LexceptSTART
+#endif
+
+ .byte 4 # DW_CFA_advance_loc4
+ .long .Lpush_r12-.LSTARTCODE
+ .byte 14 # DW_CFA_def_cfa_offset
+ .uleb128 16
+ .byte 0x8c # DW_CFA_offset %r12
+ .uleb128 2
+ .byte 4 # DW_CFA_advance_loc4
+ .long .Lpush_r13-.Lpush_r12
+ .byte 14 # DW_CFA_def_cfa_offset
+ .uleb128 24
+ .byte 0x8d # DW_CFA_offset %r13
+ .uleb128 3
+ .byte 4 # DW_CFA_advance_loc4
+ .long .Lpush_r14-.Lpush_r13
+ .byte 14 # DW_CFA_def_cfa_offset
+ .uleb128 32
+ .byte 0x8e # DW_CFA_offset %r14
+ .uleb128 4
+ .byte 4 # DW_CFA_advance_loc4
+ .long .Lsubq-.Lpush_r14
+ .byte 14 # DW_CFA_def_cfa_offset
+ .uleb128 56
+ .byte 4 # DW_CFA_advance_loc4
+ .long .Laddq-.Lsubq
+ .byte 14 # DW_CFA_def_cfa_offset
+ .uleb128 32
+ .byte 4 # DW_CFA_advance_loc4
+ .long .Lpop_r14-.Laddq
+ .byte 14 # DW_CFA_def_cfa_offset
+ .uleb128 24
+ .byte 0xce # DW_CFA_restore %r14
+ .byte 4 # DW_CFA_advance_loc4
+ .long .Lpop_r13-.Lpop_r14
+ .byte 14 # DW_CFA_def_cfa_offset
+ .uleb128 16
+ .byte 0xcd # DW_CFA_restore %r13
+ .byte 4 # DW_CFA_advance_loc4
+ .long .Lpop_r12-.Lpop_r13
+ .byte 14 # DW_CFA_def_cfa_offset
+ .uleb128 8
+ .byte 0xcc # DW_CFA_restore %r12
+ .byte 4 # DW_CFA_advance_loc4
+ .long .Lafter_retq-.Lpop_r12
+ .byte 14 # DW_CFA_def_cfa_offset
+ .uleb128 56
+ .byte 0x8c # DW_CFA_offset %r12
+ .uleb128 2
+ .byte 0x8d # DW_CFA_offset %r13
+ .uleb128 3
+ .byte 0x8e # DW_CFA_offset %r14
+ .uleb128 4
+ .align 8
+.LENDFDE:
+
+
+#ifdef SHARED
+ .hidden DW.ref.__gcc_personality_v0
+ .weak DW.ref.__gcc_personality_v0
+ .section .gnu.linkonce.d.DW.ref.__gcc_personality_v0,"aw",@progbits
+ .align 8
+ .type DW.ref.__gcc_personality_v0, @object
+ .size DW.ref.__gcc_personality_v0, 8
+DW.ref.__gcc_personality_v0:
+ .quad __gcc_personality_v0
+#endif
--- libc/nptl/sysdeps/unix/sysv/linux/x86_64/sem_trywait.S 24 Oct 2005 20:50:36 -0000 1.7
+++ libc/nptl/sysdeps/unix/sysv/linux/x86_64/sem_trywait.S 1 Aug 2007 04:36:18 -0000 1.9
@@ -18,15 +18,10 @@
02111-1307 USA. */
#include <sysdep.h>
+#include <lowlevellock.h>
#include <shlib-compat.h>
#include <pthread-errnos.h>
-#ifndef UP
-# define LOCK lock
-#else
-# define
-#endif
-
.text
.globl sem_trywait
@@ -37,7 +32,7 @@ sem_trywait:
2: testl %eax, %eax
jz 1f
- leaq -1(%rax), %rdx
+ leal -1(%rax), %edx
LOCK
cmpxchgl %edx, (%rdi)
jne 2b
--- libc/nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S 15 May 2007 06:24:23 -0000 1.10
+++ libc/nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S 1 Aug 2007 04:39:51 -0000 1.13
@@ -1,4 +1,4 @@
-/* Copyright (C) 2002, 2003, 2005 Free Software Foundation, Inc.
+/* Copyright (C) 2002, 2003, 2005, 2007 Free Software Foundation, Inc.
This file is part of the GNU C Library.
Contributed by Ulrich Drepper <drepper@redhat.com>, 2002.
@@ -18,16 +18,10 @@
02111-1307 USA. */
#include <sysdep.h>
+#include <lowlevellock.h>
#include <shlib-compat.h>
#include <pthread-errnos.h>
-
-#ifndef UP
-# define LOCK lock
-#else
-# define
-#endif
-
-#define SYS_futex 202
+#include <structsem.h>
.text
@@ -35,63 +29,93 @@
.globl sem_wait
.type sem_wait,@function
.align 16
- cfi_startproc
sem_wait:
- /* First check for cancellation. */
- movl %fs:CANCELHANDLING, %eax
- andl $0xfffffff9, %eax
- cmpl $8, %eax
- je 4f
-
+.LSTARTCODE:
pushq %r12
- cfi_adjust_cfa_offset(8)
- cfi_offset(12, -16)
+.Lpush_r12:
pushq %r13
- cfi_adjust_cfa_offset(8)
+.Lpush_r13:
movq %rdi, %r13
- cfi_offset(13, -24)
-3: movl (%r13), %eax
+#if VALUE == 0
+ movl (%r13), %eax
+#else
+ movl VALUE(%r13), %eax
+#endif
2: testl %eax, %eax
je 1f
- leaq -1(%rax), %rdx
+ leal -1(%rax), %edx
LOCK
+#if VALUE == 0
cmpxchgl %edx, (%r13)
+#else
+ cmpxchgl %edx, VALUE(%r13)
+#endif
jne 2b
- xorl %eax, %eax
- popq %r13
- cfi_adjust_cfa_offset(-8)
- cfi_restore(13)
+7: xorl %eax, %eax
+
+9: popq %r13
+.Lpop_r13:
popq %r12
- cfi_adjust_cfa_offset(-8)
- cfi_restore(12)
+.Lpop_r12:
retq
- cfi_adjust_cfa_offset(16)
- cfi_offset(12, -16)
- cfi_offset(13, -24)
-1: call __pthread_enable_asynccancel
+.Lafter_retq:
+1: LOCK
+ addq $1, NWAITERS(%r13)
+
+.LcleanupSTART:
+6: call __pthread_enable_asynccancel
movl %eax, %r8d
xorq %r10, %r10
movl $SYS_futex, %eax
movq %r13, %rdi
- movq %r10, %rsi
- movq %r10, %rdx
+#if FUTEX_WAIT == 0
+ movl PRIVATE(%rdi), %esi
+#else
+ movl $FUTEX_WAIT, %esi
+ orl PRIVATE(%rdi), %esi
+#endif
+ xorl %edx, %edx
syscall
movq %rax, %r12
movl %r8d, %edi
call __pthread_disable_asynccancel
+.LcleanupEND:
testq %r12, %r12
- je 3b
+ je 3f
cmpq $-EWOULDBLOCK, %r12
- je 3b
- negq %r12
+ jne 4f
+
+3:
+#if VALUE == 0
+ movl (%r13), %eax
+#else
+ movl VALUE(%r13), %eax
+#endif
+5: testl %eax, %eax
+ je 6b
+
+ leal -1(%rax), %edx
+ LOCK
+#if VALUE == 0
+ cmpxchgl %edx, (%r13)
+#else
+ cmpxchgl %edx, VALUE(%r13)
+#endif
+ jne 5b
+
+ LOCK
+ subq $1, NWAITERS(%r13)
+ jmp 7b
+
+4: negq %r12
#if USE___THREAD
movq errno@gottpoff(%rip), %rdx
movl %r12d, %fs:(%rdx)
@@ -101,20 +125,142 @@ sem_wait:
#endif
orl $-1, %eax
- popq %r13
- cfi_adjust_cfa_offset(-8)
- cfi_restore(13)
- popq %r12
- cfi_adjust_cfa_offset(-8)
- cfi_restore(12)
+ LOCK
+ subq $1, NWAITERS(%r13)
+
+ jmp 9b
+ .size sem_wait,.-sem_wait
- retq
-4: /* Canceled. */
- movq $0xffffffffffffffff, %fs:RESULT
+ .type sem_wait_cleanup,@function
+sem_wait_cleanup:
LOCK
- orl $0x10, %fs:CANCELHANDLING
- movq %fs:CLEANUP_JMP_BUF, %rdi
- jmp HIDDEN_JUMPTARGET (__pthread_unwind)
- cfi_endproc
- .size sem_wait,.-sem_wait
+ subq $1, NWAITERS(%r13)
+ movq %rax, %rdi
+.LcallUR:
+ call _Unwind_Resume@PLT
+ hlt
+.LENDCODE:
+ .size sem_wait_cleanup,.-sem_wait_cleanup
+
+
+ .section .gcc_except_table,"a",@progbits
+.LexceptSTART:
+ .byte 0xff # @LPStart format (omit)
+ .byte 0xff # @TType format (omit)
+ .byte 0x01 # call-site format
+ # DW_EH_PE_uleb128
+ .uleb128 .Lcstend-.Lcstbegin
+.Lcstbegin:
+ .uleb128 .LcleanupSTART-.LSTARTCODE
+ .uleb128 .LcleanupEND-.LcleanupSTART
+ .uleb128 sem_wait_cleanup-.LSTARTCODE
+ .uleb128 0
+ .uleb128 .LcallUR-.LSTARTCODE
+ .uleb128 .LENDCODE-.LcallUR
+ .uleb128 0
+ .uleb128 0
+.Lcstend:
+
+
+ .section .eh_frame,"a",@progbits
+.LSTARTFRAME:
+ .long .LENDCIE-.LSTARTCIE # Length of the CIE.
+.LSTARTCIE:
+ .long 0 # CIE ID.
+ .byte 1 # Version number.
+#ifdef SHARED
+ .string "zPLR" # NUL-terminated augmentation
+ # string.
+#else
+ .string "zPL" # NUL-terminated augmentation
+ # string.
+#endif
+ .uleb128 1 # Code alignment factor.
+ .sleb128 -8 # Data alignment factor.
+ .byte 16 # Return address register
+ # column.
+#ifdef SHARED
+ .uleb128 7 # Augmentation value length.
+ .byte 0x9b # Personality: DW_EH_PE_pcrel
+ # + DW_EH_PE_sdata4
+ # + DW_EH_PE_indirect
+ .long DW.ref.__gcc_personality_v0-.
+ .byte 0x1b # LSDA Encoding: DW_EH_PE_pcrel
+ # + DW_EH_PE_sdata4.
+ .byte 0x1b # FDE Encoding: DW_EH_PE_pcrel
+ # + DW_EH_PE_sdata4.
+#else
+ .uleb128 10 # Augmentation value length.
+ .byte 0x0 # Personality: absolute
+ .quad __gcc_personality_v0
+ .byte 0x0 # LSDA Encoding: absolute
+#endif
+ .byte 0x0c # DW_CFA_def_cfa
+ .uleb128 7
+ .uleb128 8
+ .byte 0x90 # DW_CFA_offset, column 0x10
+ .uleb128 1
+ .align 8
+.LENDCIE:
+
+ .long .LENDFDE-.LSTARTFDE # Length of the FDE.
+.LSTARTFDE:
+ .long .LSTARTFDE-.LSTARTFRAME # CIE pointer.
+#ifdef SHARED
+ .long .LSTARTCODE-. # PC-relative start address
+ # of the code.
+ .long .LENDCODE-.LSTARTCODE # Length of the code.
+ .uleb128 4 # Augmentation size
+ .long .LexceptSTART-.
+#else
+ .quad .LSTARTCODE # Start address of the code.
+ .quad .LENDCODE-.LSTARTCODE # Length of the code.
+ .uleb128 8 # Augmentation size
+ .quad .LexceptSTART
+#endif
+
+ .byte 4 # DW_CFA_advance_loc4
+ .long .Lpush_r12-.LSTARTCODE
+ .byte 14 # DW_CFA_def_cfa_offset
+ .uleb128 16
+ .byte 0x8c # DW_CFA_offset %r12
+ .uleb128 2
+ .byte 4 # DW_CFA_advance_loc4
+ .long .Lpush_r13-.Lpush_r12
+ .byte 14 # DW_CFA_def_cfa_offset
+ .uleb128 24
+ .byte 0x8d # DW_CFA_offset %r13
+ .uleb128 3
+ .byte 4 # DW_CFA_advance_loc4
+ .long .Lpop_r13-.Lpush_r13
+ .byte 14 # DW_CFA_def_cfa_offset
+ .uleb128 16
+ .byte 0xcd # DW_CFA_restore %r13
+ .byte 4 # DW_CFA_advance_loc4
+ .long .Lpop_r12-.Lpop_r13
+ .byte 14 # DW_CFA_def_cfa_offset
+ .uleb128 8
+ .byte 0xcc # DW_CFA_restore %r12
+ .byte 4 # DW_CFA_advance_loc4
+ .long .Lafter_retq-.Lpop_r12
+ .byte 14 # DW_CFA_def_cfa_offset
+ .uleb128 24
+ .byte 0x8c # DW_CFA_offset %r12
+ .uleb128 2
+ .byte 0x8d # DW_CFA_offset %r13
+ .uleb128 3
+ .align 8
+.LENDFDE:
+
+
+#ifdef SHARED
+ .hidden DW.ref.__gcc_personality_v0
+ .weak DW.ref.__gcc_personality_v0
+ .section .gnu.linkonce.d.DW.ref.__gcc_personality_v0,"aw",@progbits
+ .align 8
+ .type DW.ref.__gcc_personality_v0, @object
+ .size DW.ref.__gcc_personality_v0, 8
+DW.ref.__gcc_personality_v0:
+ .quad __gcc_personality_v0
+#endif
--- libc/nptl/sysdeps/unix/sysv/linux/x86_64/bits/pthreadtypes.h 13 Feb 2006 01:26:49 -0000 1.25
+++ libc/nptl/sysdeps/unix/sysv/linux/x86_64/bits/pthreadtypes.h 27 May 2007 18:02:09 -0000 1.26
@@ -159,9 +159,9 @@ typedef union
unsigned int __nr_readers_queued;
unsigned int __nr_writers_queued;
int __writer;
- int __pad1;
+ int __shared;
+ unsigned long int __pad1;
unsigned long int __pad2;
- unsigned long int __pad3;
/* FLAGS must stay at this position in the structure to maintain
binary compatibility. */
unsigned int __flags;
@@ -177,7 +177,10 @@ typedef union
unsigned int __nr_writers_queued;
/* FLAGS must stay at this position in the structure to maintain
binary compatibility. */
- unsigned int __flags;
+ unsigned char __flags;
+ unsigned char __shared;
+ unsigned char __pad1;
+ unsigned char __pad2;
int __writer;
} __data;
# endif
--- libc/nptl/sysdeps/x86_64/tcb-offsets.sym 26 Apr 2007 04:45:47 -0000 1.8
+++ libc/nptl/sysdeps/x86_64/tcb-offsets.sym 23 May 2007 20:49:55 -0000 1.9
@@ -11,3 +11,6 @@ CLEANUP_PREV offsetof (struct _pthread_
MUTEX_FUTEX offsetof (pthread_mutex_t, __data.__lock)
MULTIPLE_THREADS_OFFSET offsetof (tcbhead_t, multiple_threads)
POINTER_GUARD offsetof (tcbhead_t, pointer_guard)
+#ifndef __ASSUME_PRIVATE_FUTEX
+PRIVATE_FUTEX offsetof (tcbhead_t, private_futex)
+#endif
--- libc/nptl/sysdeps/x86_64/tls.h 21 May 2007 22:30:13 -0000 1.30
+++ libc/nptl/sysdeps/x86_64/tls.h 28 Jul 2007 19:30:21 -0000 1.37
@@ -51,6 +51,10 @@ typedef struct
uintptr_t sysinfo;
uintptr_t stack_guard;
uintptr_t pointer_guard;
+ unsigned long int vgetcpu_cache[2];
+#ifndef __ASSUME_PRIVATE_FUTEX
+ int private_futex;
+#endif
} tcbhead_t;
#else /* __ASSEMBLER__ */
@@ -73,6 +77,7 @@ typedef struct
#ifndef __ASSEMBLER__
/* Get system call information. */
# include <sysdep.h>
+# include <kernel-features.h>
/* Get the thread descriptor definition. */
@@ -352,7 +357,7 @@ typedef struct
: "i" (offsetof (struct pthread, header.gscope_flag)), \
"0" (THREAD_GSCOPE_FLAG_UNUSED)); \
if (__res == THREAD_GSCOPE_FLAG_WAIT) \
- lll_futex_wake (&THREAD_SELF->header.gscope_flag, 1); \
+ lll_futex_wake (&THREAD_SELF->header.gscope_flag, 1, LLL_PRIVATE); \
} \
while (0)
#define THREAD_GSCOPE_SET_FLAG() \
--- libc/nptl/sysdeps/ia64/tls.h 27 Oct 2006 23:11:43 -0000 1.12
+++ libc/nptl/sysdeps/ia64/tls.h 28 Jul 2007 19:30:20 -0000 1.15
@@ -176,7 +176,7 @@ register struct pthread *__thread_self _
= atomic_exchange_rel (&THREAD_SELF->header.gscope_flag, \
THREAD_GSCOPE_FLAG_UNUSED); \
if (__res == THREAD_GSCOPE_FLAG_WAIT) \
- lll_futex_wake (&THREAD_SELF->header.gscope_flag, 1); \
+ lll_futex_wake (&THREAD_SELF->header.gscope_flag, 1, LLL_PRIVATE); \
} \
while (0)
#define THREAD_GSCOPE_SET_FLAG() \
--- libc/nptl/sysdeps/powerpc/tls.h 27 Oct 2006 23:11:43 -0000 1.12
+++ libc/nptl/sysdeps/powerpc/tls.h 28 Jul 2007 19:30:20 -0000 1.16
@@ -193,7 +193,7 @@ register void *__thread_register __asm__
= atomic_exchange_rel (&THREAD_SELF->header.gscope_flag, \
THREAD_GSCOPE_FLAG_UNUSED); \
if (__res == THREAD_GSCOPE_FLAG_WAIT) \
- lll_futex_wake (&THREAD_SELF->header.gscope_flag, 1); \
+ lll_futex_wake (&THREAD_SELF->header.gscope_flag, 1, LLL_PRIVATE); \
} \
while (0)
#define THREAD_GSCOPE_SET_FLAG() \
--- libc/rtkaio/sysdeps/unix/sysv/linux/kaio_misc.h 2006-09-07 06:36:11.000000000 -0400
+++ libc/rtkaio/sysdeps/unix/sysv/linux/kaio_misc.h 2008-08-28 09:54:41.000000000 -0400
@@ -55,7 +55,7 @@
# define AIO_MISC_NOTIFY(waitlist) \
do { \
if (*waitlist->counterp > 0 && --*waitlist->counterp == 0) \
- lll_futex_wake (waitlist->counterp, 1); \
+ lll_futex_wake (waitlist->counterp, 1, LLL_PRIVATE); \
} while (0)
# define AIO_MISC_WAIT(result, futex, timeout, cancel) \
@@ -74,7 +74,8 @@
int status; \
do \
{ \
- status = lll_futex_timed_wait (futexaddr, oldval, timeout); \
+ status = lll_futex_timed_wait (futexaddr, oldval, timeout, \
+ LLL_PRIVATE); \
if (status != -EWOULDBLOCK) \
break; \
\
distrib
>
Scientific%20Linux
>
5x
>
x86_64
>
by-pkgid
>
bee470e21a8ce2cdd844d7d805aa403d
>
files
>
132
