2006-09-22 18:22:22 -04:00
|
|
|
/*
|
|
|
|
* IBM eServer eHCA Infiniband device driver for Linux on POWER
|
|
|
|
*
|
|
|
|
* Functions for EQs, NEQs and interrupts
|
|
|
|
*
|
|
|
|
* Authors: Heiko J Schick <schickhj@de.ibm.com>
|
|
|
|
* Khadija Souissi <souissi@de.ibm.com>
|
2007-07-09 09:30:39 -04:00
|
|
|
* Hoang-Nam Nguyen <hnguyen@de.ibm.com>
|
|
|
|
* Joachim Fenkes <fenkes@de.ibm.com>
|
2006-09-22 18:22:22 -04:00
|
|
|
*
|
|
|
|
* Copyright (c) 2005 IBM Corporation
|
|
|
|
*
|
|
|
|
* All rights reserved.
|
|
|
|
*
|
|
|
|
* This source code is distributed under a dual license of GPL v2.0 and OpenIB
|
|
|
|
* BSD.
|
|
|
|
*
|
|
|
|
* OpenIB BSD License
|
|
|
|
*
|
|
|
|
* Redistribution and use in source and binary forms, with or without
|
|
|
|
* modification, are permitted provided that the following conditions are met:
|
|
|
|
*
|
|
|
|
* Redistributions of source code must retain the above copyright notice, this
|
|
|
|
* list of conditions and the following disclaimer.
|
|
|
|
*
|
|
|
|
* Redistributions in binary form must reproduce the above copyright notice,
|
|
|
|
* this list of conditions and the following disclaimer in the documentation
|
|
|
|
* and/or other materials
|
|
|
|
* provided with the distribution.
|
|
|
|
*
|
|
|
|
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
|
|
|
|
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
|
|
|
|
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
|
|
|
|
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
|
|
|
|
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
|
|
|
|
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
|
|
|
|
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
|
|
|
|
* BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
|
|
|
|
* IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
|
|
|
|
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
|
|
|
|
* POSSIBILITY OF SUCH DAMAGE.
|
|
|
|
*/
|
|
|
|
|
include cleanup: Update gfp.h and slab.h includes to prepare for breaking implicit slab.h inclusion from percpu.h
percpu.h is included by sched.h and module.h and thus ends up being
included when building most .c files. percpu.h includes slab.h which
in turn includes gfp.h making everything defined by the two files
universally available and complicating inclusion dependencies.
percpu.h -> slab.h dependency is about to be removed. Prepare for
this change by updating users of gfp and slab facilities include those
headers directly instead of assuming availability. As this conversion
needs to touch large number of source files, the following script is
used as the basis of conversion.
http://userweb.kernel.org/~tj/misc/slabh-sweep.py
The script does the followings.
* Scan files for gfp and slab usages and update includes such that
only the necessary includes are there. ie. if only gfp is used,
gfp.h, if slab is used, slab.h.
* When the script inserts a new include, it looks at the include
blocks and try to put the new include such that its order conforms
to its surrounding. It's put in the include block which contains
core kernel includes, in the same order that the rest are ordered -
alphabetical, Christmas tree, rev-Xmas-tree or at the end if there
doesn't seem to be any matching order.
* If the script can't find a place to put a new include (mostly
because the file doesn't have fitting include block), it prints out
an error message indicating which .h file needs to be added to the
file.
The conversion was done in the following steps.
1. The initial automatic conversion of all .c files updated slightly
over 4000 files, deleting around 700 includes and adding ~480 gfp.h
and ~3000 slab.h inclusions. The script emitted errors for ~400
files.
2. Each error was manually checked. Some didn't need the inclusion,
some needed manual addition while adding it to implementation .h or
embedding .c file was more appropriate for others. This step added
inclusions to around 150 files.
3. The script was run again and the output was compared to the edits
from #2 to make sure no file was left behind.
4. Several build tests were done and a couple of problems were fixed.
e.g. lib/decompress_*.c used malloc/free() wrappers around slab
APIs requiring slab.h to be added manually.
5. The script was run on all .h files but without automatically
editing them as sprinkling gfp.h and slab.h inclusions around .h
files could easily lead to inclusion dependency hell. Most gfp.h
inclusion directives were ignored as stuff from gfp.h was usually
wildly available and often used in preprocessor macros. Each
slab.h inclusion directive was examined and added manually as
necessary.
6. percpu.h was updated not to include slab.h.
7. Build test were done on the following configurations and failures
were fixed. CONFIG_GCOV_KERNEL was turned off for all tests (as my
distributed build env didn't work with gcov compiles) and a few
more options had to be turned off depending on archs to make things
build (like ipr on powerpc/64 which failed due to missing writeq).
* x86 and x86_64 UP and SMP allmodconfig and a custom test config.
* powerpc and powerpc64 SMP allmodconfig
* sparc and sparc64 SMP allmodconfig
* ia64 SMP allmodconfig
* s390 SMP allmodconfig
* alpha SMP allmodconfig
* um on x86_64 SMP allmodconfig
8. percpu.h modifications were reverted so that it could be applied as
a separate patch and serve as bisection point.
Given the fact that I had only a couple of failures from tests on step
6, I'm fairly confident about the coverage of this conversion patch.
If there is a breakage, it's likely to be something in one of the arch
headers which should be easily discoverable easily on most builds of
the specific arch.
Signed-off-by: Tejun Heo <tj@kernel.org>
Guess-its-ok-by: Christoph Lameter <cl@linux-foundation.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Lee Schermerhorn <Lee.Schermerhorn@hp.com>
2010-03-24 04:04:11 -04:00
|
|
|
#include <linux/slab.h>
|
|
|
|
|
2006-09-22 18:22:22 -04:00
|
|
|
#include "ehca_classes.h"
|
|
|
|
#include "ehca_irq.h"
|
|
|
|
#include "ehca_iverbs.h"
|
|
|
|
#include "ehca_tools.h"
|
|
|
|
#include "hcp_if.h"
|
|
|
|
#include "hipz_fns.h"
|
2006-11-06 18:56:39 -05:00
|
|
|
#include "ipz_pt_fn.h"
|
2006-09-22 18:22:22 -04:00
|
|
|
|
2007-07-12 11:53:47 -04:00
|
|
|
#define EQE_COMPLETION_EVENT EHCA_BMASK_IBM( 1, 1)
|
|
|
|
#define EQE_CQ_QP_NUMBER EHCA_BMASK_IBM( 8, 31)
|
|
|
|
#define EQE_EE_IDENTIFIER EHCA_BMASK_IBM( 2, 7)
|
|
|
|
#define EQE_CQ_NUMBER EHCA_BMASK_IBM( 8, 31)
|
|
|
|
#define EQE_QP_NUMBER EHCA_BMASK_IBM( 8, 31)
|
|
|
|
#define EQE_QP_TOKEN EHCA_BMASK_IBM(32, 63)
|
|
|
|
#define EQE_CQ_TOKEN EHCA_BMASK_IBM(32, 63)
|
|
|
|
|
|
|
|
#define NEQE_COMPLETION_EVENT EHCA_BMASK_IBM( 1, 1)
|
|
|
|
#define NEQE_EVENT_CODE EHCA_BMASK_IBM( 2, 7)
|
|
|
|
#define NEQE_PORT_NUMBER EHCA_BMASK_IBM( 8, 15)
|
|
|
|
#define NEQE_PORT_AVAILABILITY EHCA_BMASK_IBM(16, 16)
|
|
|
|
#define NEQE_DISRUPTIVE EHCA_BMASK_IBM(16, 16)
|
2007-12-20 09:06:33 -05:00
|
|
|
#define NEQE_SPECIFIC_EVENT EHCA_BMASK_IBM(16, 23)
|
2007-07-12 11:53:47 -04:00
|
|
|
|
|
|
|
#define ERROR_DATA_LENGTH EHCA_BMASK_IBM(52, 63)
|
|
|
|
#define ERROR_DATA_TYPE EHCA_BMASK_IBM( 0, 7)
|
2006-09-22 18:22:22 -04:00
|
|
|
|
|
|
|
static void queue_comp_task(struct ehca_cq *__cq);
|
|
|
|
|
2007-07-12 11:53:47 -04:00
|
|
|
static struct ehca_comp_pool *pool;
|
2006-09-22 18:22:22 -04:00
|
|
|
|
|
|
|
static inline void comp_event_callback(struct ehca_cq *cq)
|
|
|
|
{
|
|
|
|
if (!cq->ib_cq.comp_handler)
|
|
|
|
return;
|
|
|
|
|
|
|
|
spin_lock(&cq->cb_lock);
|
|
|
|
cq->ib_cq.comp_handler(&cq->ib_cq, cq->ib_cq.cq_context);
|
|
|
|
spin_unlock(&cq->cb_lock);
|
|
|
|
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
2007-07-12 11:53:47 -04:00
|
|
|
static void print_error_data(struct ehca_shca *shca, void *data,
|
|
|
|
u64 *rblock, int length)
|
2006-09-22 18:22:22 -04:00
|
|
|
{
|
|
|
|
u64 type = EHCA_BMASK_GET(ERROR_DATA_TYPE, rblock[2]);
|
|
|
|
u64 resource = rblock[1];
|
|
|
|
|
|
|
|
switch (type) {
|
|
|
|
case 0x1: /* Queue Pair */
|
|
|
|
{
|
2007-07-12 11:53:47 -04:00
|
|
|
struct ehca_qp *qp = (struct ehca_qp *)data;
|
2006-09-22 18:22:22 -04:00
|
|
|
|
|
|
|
/* only print error data if AER is set */
|
|
|
|
if (rblock[6] == 0)
|
|
|
|
return;
|
|
|
|
|
|
|
|
ehca_err(&shca->ib_device,
|
2009-01-16 17:55:28 -05:00
|
|
|
"QP 0x%x (resource=%llx) has errors.",
|
2006-09-22 18:22:22 -04:00
|
|
|
qp->ib_qp.qp_num, resource);
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
case 0x4: /* Completion Queue */
|
|
|
|
{
|
2007-07-12 11:53:47 -04:00
|
|
|
struct ehca_cq *cq = (struct ehca_cq *)data;
|
2006-09-22 18:22:22 -04:00
|
|
|
|
|
|
|
ehca_err(&shca->ib_device,
|
2009-01-16 17:55:28 -05:00
|
|
|
"CQ 0x%x (resource=%llx) has errors.",
|
2006-09-22 18:22:22 -04:00
|
|
|
cq->cq_number, resource);
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
default:
|
|
|
|
ehca_err(&shca->ib_device,
|
2009-01-16 17:55:28 -05:00
|
|
|
"Unknown error type: %llx on %s.",
|
2006-09-22 18:22:22 -04:00
|
|
|
type, shca->ib_device.name);
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
2009-01-16 17:55:28 -05:00
|
|
|
ehca_err(&shca->ib_device, "Error data is available: %llx.", resource);
|
2006-09-22 18:22:22 -04:00
|
|
|
ehca_err(&shca->ib_device, "EHCA ----- error data begin "
|
|
|
|
"---------------------------------------------------");
|
2009-01-16 17:55:28 -05:00
|
|
|
ehca_dmp(rblock, length, "resource=%llx", resource);
|
2006-09-22 18:22:22 -04:00
|
|
|
ehca_err(&shca->ib_device, "EHCA ----- error data end "
|
|
|
|
"----------------------------------------------------");
|
|
|
|
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
|
|
|
int ehca_error_data(struct ehca_shca *shca, void *data,
|
|
|
|
u64 resource)
|
|
|
|
{
|
|
|
|
|
|
|
|
unsigned long ret;
|
|
|
|
u64 *rblock;
|
|
|
|
unsigned long block_count;
|
|
|
|
|
2007-01-09 12:04:14 -05:00
|
|
|
rblock = ehca_alloc_fw_ctrlblock(GFP_ATOMIC);
|
2006-09-22 18:22:22 -04:00
|
|
|
if (!rblock) {
|
|
|
|
ehca_err(&shca->ib_device, "Cannot allocate rblock memory.");
|
|
|
|
ret = -ENOMEM;
|
|
|
|
goto error_data1;
|
|
|
|
}
|
|
|
|
|
2006-11-06 18:56:39 -05:00
|
|
|
/* rblock must be 4K aligned and should be 4K large */
|
2006-09-22 18:22:22 -04:00
|
|
|
ret = hipz_h_error_data(shca->ipz_hca_handle,
|
|
|
|
resource,
|
|
|
|
rblock,
|
|
|
|
&block_count);
|
|
|
|
|
2006-11-06 18:56:39 -05:00
|
|
|
if (ret == H_R_STATE)
|
2006-09-22 18:22:22 -04:00
|
|
|
ehca_err(&shca->ib_device,
|
2009-01-16 17:55:28 -05:00
|
|
|
"No error data is available: %llx.", resource);
|
2006-09-22 18:22:22 -04:00
|
|
|
else if (ret == H_SUCCESS) {
|
|
|
|
int length;
|
|
|
|
|
|
|
|
length = EHCA_BMASK_GET(ERROR_DATA_LENGTH, rblock[0]);
|
|
|
|
|
2006-11-06 18:56:39 -05:00
|
|
|
if (length > EHCA_PAGESIZE)
|
|
|
|
length = EHCA_PAGESIZE;
|
2006-09-22 18:22:22 -04:00
|
|
|
|
|
|
|
print_error_data(shca, data, rblock, length);
|
2006-11-06 18:56:39 -05:00
|
|
|
} else
|
2006-09-22 18:22:22 -04:00
|
|
|
ehca_err(&shca->ib_device,
|
2009-01-16 17:55:28 -05:00
|
|
|
"Error data could not be fetched: %llx", resource);
|
2006-09-22 18:22:22 -04:00
|
|
|
|
2006-11-06 18:56:39 -05:00
|
|
|
ehca_free_fw_ctrlblock(rblock);
|
2006-09-22 18:22:22 -04:00
|
|
|
|
|
|
|
error_data1:
|
|
|
|
return ret;
|
|
|
|
|
|
|
|
}
|
|
|
|
|
2007-08-31 10:03:37 -04:00
|
|
|
static void dispatch_qp_event(struct ehca_shca *shca, struct ehca_qp *qp,
|
|
|
|
enum ib_event_type event_type)
|
2006-09-22 18:22:22 -04:00
|
|
|
{
|
|
|
|
struct ib_event event;
|
|
|
|
|
2008-07-22 17:18:07 -04:00
|
|
|
/* PATH_MIG without the QP ever having been armed is false alarm */
|
|
|
|
if (event_type == IB_EVENT_PATH_MIG && !qp->mig_armed)
|
|
|
|
return;
|
|
|
|
|
2007-07-20 10:02:18 -04:00
|
|
|
event.device = &shca->ib_device;
|
2007-08-31 10:03:37 -04:00
|
|
|
event.event = event_type;
|
2006-09-22 18:22:22 -04:00
|
|
|
|
2007-07-20 10:02:18 -04:00
|
|
|
if (qp->ext_type == EQPT_SRQ) {
|
|
|
|
if (!qp->ib_srq.event_handler)
|
|
|
|
return;
|
2006-09-22 18:22:22 -04:00
|
|
|
|
2007-07-20 10:02:18 -04:00
|
|
|
event.element.srq = &qp->ib_srq;
|
|
|
|
qp->ib_srq.event_handler(&event, qp->ib_srq.srq_context);
|
|
|
|
} else {
|
|
|
|
if (!qp->ib_qp.event_handler)
|
|
|
|
return;
|
|
|
|
|
|
|
|
event.element.qp = &qp->ib_qp;
|
|
|
|
qp->ib_qp.event_handler(&event, qp->ib_qp.qp_context);
|
|
|
|
}
|
2007-08-31 10:03:37 -04:00
|
|
|
}
|
|
|
|
|
|
|
|
static void qp_event_callback(struct ehca_shca *shca, u64 eqe,
|
|
|
|
enum ib_event_type event_type, int fatal)
|
|
|
|
{
|
|
|
|
struct ehca_qp *qp;
|
|
|
|
u32 token = EHCA_BMASK_GET(EQE_QP_TOKEN, eqe);
|
|
|
|
|
|
|
|
read_lock(&ehca_qp_idr_lock);
|
|
|
|
qp = idr_find(&ehca_qp_idr, token);
|
2008-05-07 14:35:06 -04:00
|
|
|
if (qp)
|
|
|
|
atomic_inc(&qp->nr_events);
|
2007-08-31 10:03:37 -04:00
|
|
|
read_unlock(&ehca_qp_idr_lock);
|
|
|
|
|
|
|
|
if (!qp)
|
|
|
|
return;
|
|
|
|
|
|
|
|
if (fatal)
|
|
|
|
ehca_error_data(shca, qp, qp->ipz_qp_handle.handle);
|
|
|
|
|
|
|
|
dispatch_qp_event(shca, qp, fatal && qp->ext_type == EQPT_SRQ ?
|
|
|
|
IB_EVENT_SRQ_ERR : event_type);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* eHCA only processes one WQE at a time for SRQ base QPs,
|
|
|
|
* so the last WQE has been processed as soon as the QP enters
|
|
|
|
* error state.
|
|
|
|
*/
|
|
|
|
if (fatal && qp->ext_type == EQPT_SRQBASE)
|
|
|
|
dispatch_qp_event(shca, qp, IB_EVENT_QP_LAST_WQE_REACHED);
|
2006-09-22 18:22:22 -04:00
|
|
|
|
2008-05-07 14:35:06 -04:00
|
|
|
if (atomic_dec_and_test(&qp->nr_events))
|
|
|
|
wake_up(&qp->wait_completion);
|
2006-09-22 18:22:22 -04:00
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
|
|
|
static void cq_event_callback(struct ehca_shca *shca,
|
2007-02-15 11:06:33 -05:00
|
|
|
u64 eqe)
|
2006-09-22 18:22:22 -04:00
|
|
|
{
|
|
|
|
struct ehca_cq *cq;
|
|
|
|
u32 token = EHCA_BMASK_GET(EQE_CQ_TOKEN, eqe);
|
|
|
|
|
2007-07-09 09:31:10 -04:00
|
|
|
read_lock(&ehca_cq_idr_lock);
|
2006-09-22 18:22:22 -04:00
|
|
|
cq = idr_find(&ehca_cq_idr, token);
|
2007-07-09 09:30:39 -04:00
|
|
|
if (cq)
|
|
|
|
atomic_inc(&cq->nr_events);
|
2007-07-09 09:31:10 -04:00
|
|
|
read_unlock(&ehca_cq_idr_lock);
|
2006-09-22 18:22:22 -04:00
|
|
|
|
|
|
|
if (!cq)
|
|
|
|
return;
|
|
|
|
|
|
|
|
ehca_error_data(shca, cq, cq->ipz_cq_handle.handle);
|
|
|
|
|
2007-07-09 09:30:39 -04:00
|
|
|
if (atomic_dec_and_test(&cq->nr_events))
|
|
|
|
wake_up(&cq->wait_completion);
|
|
|
|
|
2006-09-22 18:22:22 -04:00
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
|
|
|
static void parse_identifier(struct ehca_shca *shca, u64 eqe)
|
|
|
|
{
|
|
|
|
u8 identifier = EHCA_BMASK_GET(EQE_EE_IDENTIFIER, eqe);
|
|
|
|
|
|
|
|
switch (identifier) {
|
|
|
|
case 0x02: /* path migrated */
|
2007-07-20 10:02:18 -04:00
|
|
|
qp_event_callback(shca, eqe, IB_EVENT_PATH_MIG, 0);
|
2006-09-22 18:22:22 -04:00
|
|
|
break;
|
|
|
|
case 0x03: /* communication established */
|
2007-07-20 10:02:18 -04:00
|
|
|
qp_event_callback(shca, eqe, IB_EVENT_COMM_EST, 0);
|
2006-09-22 18:22:22 -04:00
|
|
|
break;
|
|
|
|
case 0x04: /* send queue drained */
|
2007-07-20 10:02:18 -04:00
|
|
|
qp_event_callback(shca, eqe, IB_EVENT_SQ_DRAINED, 0);
|
2006-09-22 18:22:22 -04:00
|
|
|
break;
|
|
|
|
case 0x05: /* QP error */
|
|
|
|
case 0x06: /* QP error */
|
2007-07-20 10:02:18 -04:00
|
|
|
qp_event_callback(shca, eqe, IB_EVENT_QP_FATAL, 1);
|
2006-09-22 18:22:22 -04:00
|
|
|
break;
|
|
|
|
case 0x07: /* CQ error */
|
|
|
|
case 0x08: /* CQ error */
|
|
|
|
cq_event_callback(shca, eqe);
|
|
|
|
break;
|
|
|
|
case 0x09: /* MRMWPTE error */
|
|
|
|
ehca_err(&shca->ib_device, "MRMWPTE error.");
|
|
|
|
break;
|
|
|
|
case 0x0A: /* port event */
|
|
|
|
ehca_err(&shca->ib_device, "Port event.");
|
|
|
|
break;
|
|
|
|
case 0x0B: /* MR access error */
|
|
|
|
ehca_err(&shca->ib_device, "MR access error.");
|
|
|
|
break;
|
|
|
|
case 0x0C: /* EQ error */
|
|
|
|
ehca_err(&shca->ib_device, "EQ error.");
|
|
|
|
break;
|
|
|
|
case 0x0D: /* P/Q_Key mismatch */
|
|
|
|
ehca_err(&shca->ib_device, "P/Q_Key mismatch.");
|
|
|
|
break;
|
|
|
|
case 0x10: /* sampling complete */
|
|
|
|
ehca_err(&shca->ib_device, "Sampling complete.");
|
|
|
|
break;
|
|
|
|
case 0x11: /* unaffiliated access error */
|
|
|
|
ehca_err(&shca->ib_device, "Unaffiliated access error.");
|
|
|
|
break;
|
2007-09-11 09:34:04 -04:00
|
|
|
case 0x12: /* path migrating */
|
|
|
|
ehca_err(&shca->ib_device, "Path migrating.");
|
2006-09-22 18:22:22 -04:00
|
|
|
break;
|
|
|
|
case 0x13: /* interface trace stopped */
|
|
|
|
ehca_err(&shca->ib_device, "Interface trace stopped.");
|
|
|
|
break;
|
|
|
|
case 0x14: /* first error capture info available */
|
2007-07-20 10:02:18 -04:00
|
|
|
ehca_info(&shca->ib_device, "First error capture available");
|
|
|
|
break;
|
|
|
|
case 0x15: /* SRQ limit reached */
|
|
|
|
qp_event_callback(shca, eqe, IB_EVENT_SRQ_LIMIT_REACHED, 0);
|
|
|
|
break;
|
2006-09-22 18:22:22 -04:00
|
|
|
default:
|
|
|
|
ehca_err(&shca->ib_device, "Unknown identifier: %x on %s.",
|
|
|
|
identifier, shca->ib_device.name);
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
2007-07-09 09:32:22 -04:00
|
|
|
static void dispatch_port_event(struct ehca_shca *shca, int port_num,
|
|
|
|
enum ib_event_type type, const char *msg)
|
2006-09-22 18:22:22 -04:00
|
|
|
{
|
|
|
|
struct ib_event event;
|
2007-07-09 09:32:22 -04:00
|
|
|
|
|
|
|
ehca_info(&shca->ib_device, "port %d %s.", port_num, msg);
|
|
|
|
event.device = &shca->ib_device;
|
|
|
|
event.event = type;
|
|
|
|
event.element.port_num = port_num;
|
|
|
|
ib_dispatch_event(&event);
|
|
|
|
}
|
|
|
|
|
|
|
|
static void notify_port_conf_change(struct ehca_shca *shca, int port_num)
|
|
|
|
{
|
|
|
|
struct ehca_sma_attr new_attr;
|
|
|
|
struct ehca_sma_attr *old_attr = &shca->sport[port_num - 1].saved_attr;
|
|
|
|
|
|
|
|
ehca_query_sma_attr(shca, port_num, &new_attr);
|
|
|
|
|
|
|
|
if (new_attr.sm_sl != old_attr->sm_sl ||
|
|
|
|
new_attr.sm_lid != old_attr->sm_lid)
|
|
|
|
dispatch_port_event(shca, port_num, IB_EVENT_SM_CHANGE,
|
|
|
|
"SM changed");
|
|
|
|
|
|
|
|
if (new_attr.lid != old_attr->lid ||
|
|
|
|
new_attr.lmc != old_attr->lmc)
|
|
|
|
dispatch_port_event(shca, port_num, IB_EVENT_LID_CHANGE,
|
|
|
|
"LID changed");
|
|
|
|
|
|
|
|
if (new_attr.pkey_tbl_len != old_attr->pkey_tbl_len ||
|
|
|
|
memcmp(new_attr.pkeys, old_attr->pkeys,
|
|
|
|
sizeof(u16) * new_attr.pkey_tbl_len))
|
|
|
|
dispatch_port_event(shca, port_num, IB_EVENT_PKEY_CHANGE,
|
|
|
|
"P_Key changed");
|
|
|
|
|
|
|
|
*old_attr = new_attr;
|
|
|
|
}
|
|
|
|
|
2008-11-11 18:44:22 -05:00
|
|
|
/* replay modify_qp for sqps -- return 0 if all is well, 1 if AQP1 destroyed */
|
|
|
|
static int replay_modify_qp(struct ehca_sport *sport)
|
|
|
|
{
|
|
|
|
int aqp1_destroyed;
|
|
|
|
unsigned long flags;
|
|
|
|
|
|
|
|
spin_lock_irqsave(&sport->mod_sqp_lock, flags);
|
|
|
|
|
|
|
|
aqp1_destroyed = !sport->ibqp_sqp[IB_QPT_GSI];
|
|
|
|
|
|
|
|
if (sport->ibqp_sqp[IB_QPT_SMI])
|
|
|
|
ehca_recover_sqp(sport->ibqp_sqp[IB_QPT_SMI]);
|
|
|
|
if (!aqp1_destroyed)
|
|
|
|
ehca_recover_sqp(sport->ibqp_sqp[IB_QPT_GSI]);
|
|
|
|
|
|
|
|
spin_unlock_irqrestore(&sport->mod_sqp_lock, flags);
|
|
|
|
|
|
|
|
return aqp1_destroyed;
|
|
|
|
}
|
|
|
|
|
2007-07-09 09:32:22 -04:00
|
|
|
static void parse_ec(struct ehca_shca *shca, u64 eqe)
|
|
|
|
{
|
2006-09-22 18:22:22 -04:00
|
|
|
u8 ec = EHCA_BMASK_GET(NEQE_EVENT_CODE, eqe);
|
|
|
|
u8 port = EHCA_BMASK_GET(NEQE_PORT_NUMBER, eqe);
|
2007-12-20 09:06:33 -05:00
|
|
|
u8 spec_event;
|
2008-01-17 09:05:45 -05:00
|
|
|
struct ehca_sport *sport = &shca->sport[port - 1];
|
2006-09-22 18:22:22 -04:00
|
|
|
|
|
|
|
switch (ec) {
|
|
|
|
case 0x30: /* port availability change */
|
|
|
|
if (EHCA_BMASK_GET(NEQE_PORT_AVAILABILITY, eqe)) {
|
2008-11-11 18:44:22 -05:00
|
|
|
/* only replay modify_qp calls in autodetect mode;
|
|
|
|
* if AQP1 was destroyed, the port is already down
|
|
|
|
* again and we can drop the event.
|
|
|
|
*/
|
|
|
|
if (ehca_nr_ports < 0)
|
|
|
|
if (replay_modify_qp(sport))
|
|
|
|
break;
|
2008-01-17 09:05:45 -05:00
|
|
|
|
|
|
|
sport->port_state = IB_PORT_ACTIVE;
|
2007-07-09 09:32:22 -04:00
|
|
|
dispatch_port_event(shca, port, IB_EVENT_PORT_ACTIVE,
|
|
|
|
"is active");
|
2008-11-11 18:44:22 -05:00
|
|
|
ehca_query_sma_attr(shca, port, &sport->saved_attr);
|
2006-09-22 18:22:22 -04:00
|
|
|
} else {
|
2008-01-17 09:05:45 -05:00
|
|
|
sport->port_state = IB_PORT_DOWN;
|
2007-07-09 09:32:22 -04:00
|
|
|
dispatch_port_event(shca, port, IB_EVENT_PORT_ERR,
|
|
|
|
"is inactive");
|
2006-09-22 18:22:22 -04:00
|
|
|
}
|
|
|
|
break;
|
|
|
|
case 0x31:
|
|
|
|
/* port configuration change
|
|
|
|
* disruptive change is caused by
|
|
|
|
* LID, PKEY or SM change
|
|
|
|
*/
|
2007-07-09 09:32:22 -04:00
|
|
|
if (EHCA_BMASK_GET(NEQE_DISRUPTIVE, eqe)) {
|
|
|
|
ehca_warn(&shca->ib_device, "disruptive port "
|
|
|
|
"%d configuration change", port);
|
|
|
|
|
2008-01-17 09:05:45 -05:00
|
|
|
sport->port_state = IB_PORT_DOWN;
|
2007-07-09 09:32:22 -04:00
|
|
|
dispatch_port_event(shca, port, IB_EVENT_PORT_ERR,
|
|
|
|
"is inactive");
|
|
|
|
|
2008-01-17 09:05:45 -05:00
|
|
|
sport->port_state = IB_PORT_ACTIVE;
|
2007-07-09 09:32:22 -04:00
|
|
|
dispatch_port_event(shca, port, IB_EVENT_PORT_ACTIVE,
|
|
|
|
"is active");
|
2008-01-25 15:12:39 -05:00
|
|
|
ehca_query_sma_attr(shca, port,
|
|
|
|
&sport->saved_attr);
|
2007-07-09 09:32:22 -04:00
|
|
|
} else
|
|
|
|
notify_port_conf_change(shca, port);
|
2006-09-22 18:22:22 -04:00
|
|
|
break;
|
|
|
|
case 0x32: /* adapter malfunction */
|
|
|
|
ehca_err(&shca->ib_device, "Adapter malfunction.");
|
|
|
|
break;
|
|
|
|
case 0x33: /* trace stopped */
|
|
|
|
ehca_err(&shca->ib_device, "Traced stopped.");
|
|
|
|
break;
|
2007-12-20 09:06:33 -05:00
|
|
|
case 0x34: /* util async event */
|
|
|
|
spec_event = EHCA_BMASK_GET(NEQE_SPECIFIC_EVENT, eqe);
|
|
|
|
if (spec_event == 0x80) /* client reregister required */
|
|
|
|
dispatch_port_event(shca, port,
|
|
|
|
IB_EVENT_CLIENT_REREGISTER,
|
|
|
|
"client reregister req.");
|
|
|
|
else
|
|
|
|
ehca_warn(&shca->ib_device, "Unknown util async "
|
|
|
|
"event %x on port %x", spec_event, port);
|
|
|
|
break;
|
2006-09-22 18:22:22 -04:00
|
|
|
default:
|
|
|
|
ehca_err(&shca->ib_device, "Unknown event code: %x on %s.",
|
|
|
|
ec, shca->ib_device.name);
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline void reset_eq_pending(struct ehca_cq *cq)
|
|
|
|
{
|
|
|
|
u64 CQx_EP;
|
|
|
|
struct h_galpa gal = cq->galpas.kernel;
|
|
|
|
|
|
|
|
hipz_galpa_store_cq(gal, cqx_ep, 0x0);
|
|
|
|
CQx_EP = hipz_galpa_load(gal, CQTEMM_OFFSET(cqx_ep));
|
|
|
|
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
IRQ: Maintain regs pointer globally rather than passing to IRQ handlers
Maintain a per-CPU global "struct pt_regs *" variable which can be used instead
of passing regs around manually through all ~1800 interrupt handlers in the
Linux kernel.
The regs pointer is used in few places, but it potentially costs both stack
space and code to pass it around. On the FRV arch, removing the regs parameter
from all the genirq function results in a 20% speed up of the IRQ exit path
(ie: from leaving timer_interrupt() to leaving do_IRQ()).
Where appropriate, an arch may override the generic storage facility and do
something different with the variable. On FRV, for instance, the address is
maintained in GR28 at all times inside the kernel as part of general exception
handling.
Having looked over the code, it appears that the parameter may be handed down
through up to twenty or so layers of functions. Consider a USB character
device attached to a USB hub, attached to a USB controller that posts its
interrupts through a cascaded auxiliary interrupt controller. A character
device driver may want to pass regs to the sysrq handler through the input
layer which adds another few layers of parameter passing.
I've build this code with allyesconfig for x86_64 and i386. I've runtested the
main part of the code on FRV and i386, though I can't test most of the drivers.
I've also done partial conversion for powerpc and MIPS - these at least compile
with minimal configurations.
This will affect all archs. Mostly the changes should be relatively easy.
Take do_IRQ(), store the regs pointer at the beginning, saving the old one:
struct pt_regs *old_regs = set_irq_regs(regs);
And put the old one back at the end:
set_irq_regs(old_regs);
Don't pass regs through to generic_handle_irq() or __do_IRQ().
In timer_interrupt(), this sort of change will be necessary:
- update_process_times(user_mode(regs));
- profile_tick(CPU_PROFILING, regs);
+ update_process_times(user_mode(get_irq_regs()));
+ profile_tick(CPU_PROFILING);
I'd like to move update_process_times()'s use of get_irq_regs() into itself,
except that i386, alone of the archs, uses something other than user_mode().
Some notes on the interrupt handling in the drivers:
(*) input_dev() is now gone entirely. The regs pointer is no longer stored in
the input_dev struct.
(*) finish_unlinks() in drivers/usb/host/ohci-q.c needs checking. It does
something different depending on whether it's been supplied with a regs
pointer or not.
(*) Various IRQ handler function pointers have been moved to type
irq_handler_t.
Signed-Off-By: David Howells <dhowells@redhat.com>
(cherry picked from 1b16e7ac850969f38b375e511e3fa2f474a33867 commit)
2006-10-05 09:55:46 -04:00
|
|
|
irqreturn_t ehca_interrupt_neq(int irq, void *dev_id)
|
2006-09-22 18:22:22 -04:00
|
|
|
{
|
|
|
|
struct ehca_shca *shca = (struct ehca_shca*)dev_id;
|
|
|
|
|
|
|
|
tasklet_hi_schedule(&shca->neq.interrupt_task);
|
|
|
|
|
|
|
|
return IRQ_HANDLED;
|
|
|
|
}
|
|
|
|
|
|
|
|
void ehca_tasklet_neq(unsigned long data)
|
|
|
|
{
|
|
|
|
struct ehca_shca *shca = (struct ehca_shca*)data;
|
|
|
|
struct ehca_eqe *eqe;
|
|
|
|
u64 ret;
|
|
|
|
|
2009-05-13 19:53:39 -04:00
|
|
|
eqe = ehca_poll_eq(shca, &shca->neq);
|
2006-09-22 18:22:22 -04:00
|
|
|
|
|
|
|
while (eqe) {
|
|
|
|
if (!EHCA_BMASK_GET(NEQE_COMPLETION_EVENT, eqe->entry))
|
|
|
|
parse_ec(shca, eqe->entry);
|
|
|
|
|
2009-05-13 19:53:39 -04:00
|
|
|
eqe = ehca_poll_eq(shca, &shca->neq);
|
2006-09-22 18:22:22 -04:00
|
|
|
}
|
|
|
|
|
|
|
|
ret = hipz_h_reset_event(shca->ipz_hca_handle,
|
|
|
|
shca->neq.ipz_eq_handle, 0xFFFFFFFFFFFFFFFFL);
|
|
|
|
|
|
|
|
if (ret != H_SUCCESS)
|
|
|
|
ehca_err(&shca->ib_device, "Can't clear notification events.");
|
|
|
|
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
IRQ: Maintain regs pointer globally rather than passing to IRQ handlers
Maintain a per-CPU global "struct pt_regs *" variable which can be used instead
of passing regs around manually through all ~1800 interrupt handlers in the
Linux kernel.
The regs pointer is used in few places, but it potentially costs both stack
space and code to pass it around. On the FRV arch, removing the regs parameter
from all the genirq function results in a 20% speed up of the IRQ exit path
(ie: from leaving timer_interrupt() to leaving do_IRQ()).
Where appropriate, an arch may override the generic storage facility and do
something different with the variable. On FRV, for instance, the address is
maintained in GR28 at all times inside the kernel as part of general exception
handling.
Having looked over the code, it appears that the parameter may be handed down
through up to twenty or so layers of functions. Consider a USB character
device attached to a USB hub, attached to a USB controller that posts its
interrupts through a cascaded auxiliary interrupt controller. A character
device driver may want to pass regs to the sysrq handler through the input
layer which adds another few layers of parameter passing.
I've build this code with allyesconfig for x86_64 and i386. I've runtested the
main part of the code on FRV and i386, though I can't test most of the drivers.
I've also done partial conversion for powerpc and MIPS - these at least compile
with minimal configurations.
This will affect all archs. Mostly the changes should be relatively easy.
Take do_IRQ(), store the regs pointer at the beginning, saving the old one:
struct pt_regs *old_regs = set_irq_regs(regs);
And put the old one back at the end:
set_irq_regs(old_regs);
Don't pass regs through to generic_handle_irq() or __do_IRQ().
In timer_interrupt(), this sort of change will be necessary:
- update_process_times(user_mode(regs));
- profile_tick(CPU_PROFILING, regs);
+ update_process_times(user_mode(get_irq_regs()));
+ profile_tick(CPU_PROFILING);
I'd like to move update_process_times()'s use of get_irq_regs() into itself,
except that i386, alone of the archs, uses something other than user_mode().
Some notes on the interrupt handling in the drivers:
(*) input_dev() is now gone entirely. The regs pointer is no longer stored in
the input_dev struct.
(*) finish_unlinks() in drivers/usb/host/ohci-q.c needs checking. It does
something different depending on whether it's been supplied with a regs
pointer or not.
(*) Various IRQ handler function pointers have been moved to type
irq_handler_t.
Signed-Off-By: David Howells <dhowells@redhat.com>
(cherry picked from 1b16e7ac850969f38b375e511e3fa2f474a33867 commit)
2006-10-05 09:55:46 -04:00
|
|
|
irqreturn_t ehca_interrupt_eq(int irq, void *dev_id)
|
2006-09-22 18:22:22 -04:00
|
|
|
{
|
|
|
|
struct ehca_shca *shca = (struct ehca_shca*)dev_id;
|
|
|
|
|
|
|
|
tasklet_hi_schedule(&shca->eq.interrupt_task);
|
|
|
|
|
|
|
|
return IRQ_HANDLED;
|
|
|
|
}
|
|
|
|
|
|
|
|
|
2007-02-15 11:06:33 -05:00
|
|
|
static inline void process_eqe(struct ehca_shca *shca, struct ehca_eqe *eqe)
|
|
|
|
{
|
|
|
|
u64 eqe_value;
|
|
|
|
u32 token;
|
|
|
|
struct ehca_cq *cq;
|
2007-02-28 12:01:02 -05:00
|
|
|
|
2007-02-15 11:06:33 -05:00
|
|
|
eqe_value = eqe->entry;
|
2009-01-16 17:55:28 -05:00
|
|
|
ehca_dbg(&shca->ib_device, "eqe_value=%llx", eqe_value);
|
2007-02-15 11:06:33 -05:00
|
|
|
if (EHCA_BMASK_GET(EQE_COMPLETION_EVENT, eqe_value)) {
|
2007-02-28 12:01:02 -05:00
|
|
|
ehca_dbg(&shca->ib_device, "Got completion event");
|
2007-02-15 11:06:33 -05:00
|
|
|
token = EHCA_BMASK_GET(EQE_CQ_TOKEN, eqe_value);
|
2007-07-09 09:31:10 -04:00
|
|
|
read_lock(&ehca_cq_idr_lock);
|
2007-02-15 11:06:33 -05:00
|
|
|
cq = idr_find(&ehca_cq_idr, token);
|
2007-07-09 09:30:39 -04:00
|
|
|
if (cq)
|
|
|
|
atomic_inc(&cq->nr_events);
|
2007-07-09 09:31:10 -04:00
|
|
|
read_unlock(&ehca_cq_idr_lock);
|
2007-02-15 11:06:33 -05:00
|
|
|
if (cq == NULL) {
|
|
|
|
ehca_err(&shca->ib_device,
|
|
|
|
"Invalid eqe for non-existing cq token=%x",
|
|
|
|
token);
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
reset_eq_pending(cq);
|
2007-02-28 12:01:02 -05:00
|
|
|
if (ehca_scaling_code)
|
2007-02-15 11:08:33 -05:00
|
|
|
queue_comp_task(cq);
|
2007-02-28 12:01:02 -05:00
|
|
|
else {
|
2007-02-15 11:08:33 -05:00
|
|
|
comp_event_callback(cq);
|
2007-07-09 09:30:39 -04:00
|
|
|
if (atomic_dec_and_test(&cq->nr_events))
|
2007-02-28 12:01:02 -05:00
|
|
|
wake_up(&cq->wait_completion);
|
2007-02-15 11:08:33 -05:00
|
|
|
}
|
2007-02-15 11:06:33 -05:00
|
|
|
} else {
|
2007-02-28 12:01:02 -05:00
|
|
|
ehca_dbg(&shca->ib_device, "Got non completion event");
|
2007-02-15 11:06:33 -05:00
|
|
|
parse_identifier(shca, eqe_value);
|
|
|
|
}
|
|
|
|
}
|
2006-09-22 18:22:22 -04:00
|
|
|
|
2007-02-15 11:06:33 -05:00
|
|
|
void ehca_process_eq(struct ehca_shca *shca, int is_irq)
|
|
|
|
{
|
|
|
|
struct ehca_eq *eq = &shca->eq;
|
|
|
|
struct ehca_eqe_cache_entry *eqe_cache = eq->eqe_cache;
|
2008-07-15 02:48:47 -04:00
|
|
|
u64 eqe_value, ret;
|
2007-02-15 11:06:33 -05:00
|
|
|
int eqe_cnt, i;
|
|
|
|
int eq_empty = 0;
|
|
|
|
|
2010-01-27 12:01:56 -05:00
|
|
|
spin_lock(&eq->irq_spinlock);
|
2007-02-15 11:06:33 -05:00
|
|
|
if (is_irq) {
|
|
|
|
const int max_query_cnt = 100;
|
|
|
|
int query_cnt = 0;
|
|
|
|
int int_state = 1;
|
|
|
|
do {
|
|
|
|
int_state = hipz_h_query_int_state(
|
|
|
|
shca->ipz_hca_handle, eq->ist);
|
|
|
|
query_cnt++;
|
|
|
|
iosync();
|
|
|
|
} while (int_state && query_cnt < max_query_cnt);
|
|
|
|
if (unlikely((query_cnt == max_query_cnt)))
|
|
|
|
ehca_dbg(&shca->ib_device, "int_state=%x query_cnt=%x",
|
|
|
|
int_state, query_cnt);
|
|
|
|
}
|
2006-09-22 18:22:22 -04:00
|
|
|
|
2007-02-15 11:06:33 -05:00
|
|
|
/* read out all eqes */
|
|
|
|
eqe_cnt = 0;
|
|
|
|
do {
|
|
|
|
u32 token;
|
2009-05-13 19:53:39 -04:00
|
|
|
eqe_cache[eqe_cnt].eqe = ehca_poll_eq(shca, eq);
|
2007-02-15 11:06:33 -05:00
|
|
|
if (!eqe_cache[eqe_cnt].eqe)
|
|
|
|
break;
|
|
|
|
eqe_value = eqe_cache[eqe_cnt].eqe->entry;
|
|
|
|
if (EHCA_BMASK_GET(EQE_COMPLETION_EVENT, eqe_value)) {
|
|
|
|
token = EHCA_BMASK_GET(EQE_CQ_TOKEN, eqe_value);
|
2007-07-09 09:31:10 -04:00
|
|
|
read_lock(&ehca_cq_idr_lock);
|
2007-02-15 11:06:33 -05:00
|
|
|
eqe_cache[eqe_cnt].cq = idr_find(&ehca_cq_idr, token);
|
2007-07-09 09:30:39 -04:00
|
|
|
if (eqe_cache[eqe_cnt].cq)
|
|
|
|
atomic_inc(&eqe_cache[eqe_cnt].cq->nr_events);
|
2007-07-09 09:31:10 -04:00
|
|
|
read_unlock(&ehca_cq_idr_lock);
|
2007-02-15 11:06:33 -05:00
|
|
|
if (!eqe_cache[eqe_cnt].cq) {
|
|
|
|
ehca_err(&shca->ib_device,
|
|
|
|
"Invalid eqe for non-existing cq "
|
|
|
|
"token=%x", token);
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
} else
|
|
|
|
eqe_cache[eqe_cnt].cq = NULL;
|
|
|
|
eqe_cnt++;
|
|
|
|
} while (eqe_cnt < EHCA_EQE_CACHE_SIZE);
|
|
|
|
if (!eqe_cnt) {
|
|
|
|
if (is_irq)
|
|
|
|
ehca_dbg(&shca->ib_device,
|
|
|
|
"No eqe found for irq event");
|
|
|
|
goto unlock_irq_spinlock;
|
2008-07-15 02:48:47 -04:00
|
|
|
} else if (!is_irq) {
|
|
|
|
ret = hipz_h_eoi(eq->ist);
|
|
|
|
if (ret != H_SUCCESS)
|
|
|
|
ehca_err(&shca->ib_device,
|
2009-01-16 17:55:28 -05:00
|
|
|
"bad return code EOI -rc = %lld\n", ret);
|
2007-02-15 11:06:33 -05:00
|
|
|
ehca_dbg(&shca->ib_device, "deadman found %x eqe", eqe_cnt);
|
2008-07-15 02:48:47 -04:00
|
|
|
}
|
2007-02-15 11:06:33 -05:00
|
|
|
if (unlikely(eqe_cnt == EHCA_EQE_CACHE_SIZE))
|
|
|
|
ehca_dbg(&shca->ib_device, "too many eqes for one irq event");
|
|
|
|
/* enable irq for new packets */
|
|
|
|
for (i = 0; i < eqe_cnt; i++) {
|
|
|
|
if (eq->eqe_cache[i].cq)
|
|
|
|
reset_eq_pending(eq->eqe_cache[i].cq);
|
|
|
|
}
|
|
|
|
/* check eq */
|
|
|
|
spin_lock(&eq->spinlock);
|
|
|
|
eq_empty = (!ipz_eqit_eq_peek_valid(&shca->eq.ipz_queue));
|
|
|
|
spin_unlock(&eq->spinlock);
|
|
|
|
/* call completion handler for cached eqes */
|
|
|
|
for (i = 0; i < eqe_cnt; i++)
|
|
|
|
if (eq->eqe_cache[i].cq) {
|
2007-02-28 12:01:02 -05:00
|
|
|
if (ehca_scaling_code)
|
2007-02-15 11:08:33 -05:00
|
|
|
queue_comp_task(eq->eqe_cache[i].cq);
|
2007-02-28 12:01:02 -05:00
|
|
|
else {
|
|
|
|
struct ehca_cq *cq = eq->eqe_cache[i].cq;
|
|
|
|
comp_event_callback(cq);
|
2007-07-09 09:30:39 -04:00
|
|
|
if (atomic_dec_and_test(&cq->nr_events))
|
2007-02-28 12:01:02 -05:00
|
|
|
wake_up(&cq->wait_completion);
|
|
|
|
}
|
2007-02-15 11:06:33 -05:00
|
|
|
} else {
|
|
|
|
ehca_dbg(&shca->ib_device, "Got non completion event");
|
|
|
|
parse_identifier(shca, eq->eqe_cache[i].eqe->entry);
|
2006-09-22 18:22:22 -04:00
|
|
|
}
|
2007-02-15 11:06:33 -05:00
|
|
|
/* poll eq if not empty */
|
|
|
|
if (eq_empty)
|
|
|
|
goto unlock_irq_spinlock;
|
|
|
|
do {
|
|
|
|
struct ehca_eqe *eqe;
|
2009-05-13 19:53:39 -04:00
|
|
|
eqe = ehca_poll_eq(shca, &shca->eq);
|
2007-02-15 11:06:33 -05:00
|
|
|
if (!eqe)
|
|
|
|
break;
|
|
|
|
process_eqe(shca, eqe);
|
|
|
|
} while (1);
|
|
|
|
|
|
|
|
unlock_irq_spinlock:
|
2010-01-27 12:01:56 -05:00
|
|
|
spin_unlock(&eq->irq_spinlock);
|
2007-02-15 11:06:33 -05:00
|
|
|
}
|
2006-09-22 18:22:22 -04:00
|
|
|
|
2007-02-15 11:06:33 -05:00
|
|
|
void ehca_tasklet_eq(unsigned long data)
|
|
|
|
{
|
|
|
|
ehca_process_eq((struct ehca_shca*)data, 1);
|
2006-09-22 18:22:22 -04:00
|
|
|
}
|
|
|
|
|
2007-07-12 11:53:47 -04:00
|
|
|
static inline int find_next_online_cpu(struct ehca_comp_pool *pool)
|
2006-09-22 18:22:22 -04:00
|
|
|
{
|
2007-02-15 11:07:30 -05:00
|
|
|
int cpu;
|
|
|
|
unsigned long flags;
|
2006-09-22 18:22:22 -04:00
|
|
|
|
2007-02-15 11:07:30 -05:00
|
|
|
WARN_ON_ONCE(!in_interrupt());
|
2008-04-23 14:55:45 -04:00
|
|
|
if (ehca_debug_level >= 3)
|
2008-12-29 17:35:18 -05:00
|
|
|
ehca_dmp(cpu_online_mask, cpumask_size(), "");
|
2006-09-22 18:22:22 -04:00
|
|
|
|
2007-02-15 11:07:30 -05:00
|
|
|
spin_lock_irqsave(&pool->last_cpu_lock, flags);
|
2008-12-29 17:35:18 -05:00
|
|
|
cpu = cpumask_next(pool->last_cpu, cpu_online_mask);
|
2008-05-12 15:21:13 -04:00
|
|
|
if (cpu >= nr_cpu_ids)
|
2008-12-29 17:35:18 -05:00
|
|
|
cpu = cpumask_first(cpu_online_mask);
|
2007-02-15 11:07:30 -05:00
|
|
|
pool->last_cpu = cpu;
|
|
|
|
spin_unlock_irqrestore(&pool->last_cpu_lock, flags);
|
2006-09-22 18:22:22 -04:00
|
|
|
|
2007-02-15 11:07:30 -05:00
|
|
|
return cpu;
|
2006-09-22 18:22:22 -04:00
|
|
|
}
|
|
|
|
|
|
|
|
static void __queue_comp_task(struct ehca_cq *__cq,
|
|
|
|
struct ehca_cpu_comp_task *cct)
|
|
|
|
{
|
2007-02-15 11:07:30 -05:00
|
|
|
unsigned long flags;
|
2006-09-22 18:22:22 -04:00
|
|
|
|
2007-02-15 11:07:30 -05:00
|
|
|
spin_lock_irqsave(&cct->task_lock, flags);
|
|
|
|
spin_lock(&__cq->task_lock);
|
2006-09-22 18:22:22 -04:00
|
|
|
|
|
|
|
if (__cq->nr_callbacks == 0) {
|
|
|
|
__cq->nr_callbacks++;
|
|
|
|
list_add_tail(&__cq->entry, &cct->cq_list);
|
|
|
|
cct->cq_jobs++;
|
|
|
|
wake_up(&cct->wait_queue);
|
2007-02-28 12:01:02 -05:00
|
|
|
} else
|
2006-09-22 18:22:22 -04:00
|
|
|
__cq->nr_callbacks++;
|
|
|
|
|
2007-02-15 11:07:30 -05:00
|
|
|
spin_unlock(&__cq->task_lock);
|
|
|
|
spin_unlock_irqrestore(&cct->task_lock, flags);
|
2006-09-22 18:22:22 -04:00
|
|
|
}
|
|
|
|
|
|
|
|
static void queue_comp_task(struct ehca_cq *__cq)
|
|
|
|
{
|
|
|
|
int cpu_id;
|
|
|
|
struct ehca_cpu_comp_task *cct;
|
2007-02-28 12:01:02 -05:00
|
|
|
int cq_jobs;
|
|
|
|
unsigned long flags;
|
2006-09-22 18:22:22 -04:00
|
|
|
|
|
|
|
cpu_id = find_next_online_cpu(pool);
|
|
|
|
BUG_ON(!cpu_online(cpu_id));
|
|
|
|
|
|
|
|
cct = per_cpu_ptr(pool->cpu_comp_tasks, cpu_id);
|
2007-02-15 11:07:30 -05:00
|
|
|
BUG_ON(!cct);
|
2006-09-22 18:22:22 -04:00
|
|
|
|
2007-02-28 12:01:02 -05:00
|
|
|
spin_lock_irqsave(&cct->task_lock, flags);
|
|
|
|
cq_jobs = cct->cq_jobs;
|
|
|
|
spin_unlock_irqrestore(&cct->task_lock, flags);
|
|
|
|
if (cq_jobs > 0) {
|
2006-09-22 18:22:22 -04:00
|
|
|
cpu_id = find_next_online_cpu(pool);
|
|
|
|
cct = per_cpu_ptr(pool->cpu_comp_tasks, cpu_id);
|
2007-02-15 11:07:30 -05:00
|
|
|
BUG_ON(!cct);
|
2006-09-22 18:22:22 -04:00
|
|
|
}
|
|
|
|
|
|
|
|
__queue_comp_task(__cq, cct);
|
|
|
|
}
|
|
|
|
|
2007-07-12 11:53:47 -04:00
|
|
|
static void run_comp_task(struct ehca_cpu_comp_task *cct)
|
2006-09-22 18:22:22 -04:00
|
|
|
{
|
|
|
|
struct ehca_cq *cq;
|
2007-02-15 11:07:30 -05:00
|
|
|
unsigned long flags;
|
2006-09-22 18:22:22 -04:00
|
|
|
|
2007-02-15 11:07:30 -05:00
|
|
|
spin_lock_irqsave(&cct->task_lock, flags);
|
2006-09-22 18:22:22 -04:00
|
|
|
|
|
|
|
while (!list_empty(&cct->cq_list)) {
|
|
|
|
cq = list_entry(cct->cq_list.next, struct ehca_cq, entry);
|
2007-02-15 11:07:30 -05:00
|
|
|
spin_unlock_irqrestore(&cct->task_lock, flags);
|
2006-09-22 18:22:22 -04:00
|
|
|
|
2007-07-09 09:30:39 -04:00
|
|
|
comp_event_callback(cq);
|
|
|
|
if (atomic_dec_and_test(&cq->nr_events))
|
2007-02-28 12:01:02 -05:00
|
|
|
wake_up(&cq->wait_completion);
|
|
|
|
|
|
|
|
spin_lock_irqsave(&cct->task_lock, flags);
|
2007-02-15 11:07:30 -05:00
|
|
|
spin_lock(&cq->task_lock);
|
2006-09-22 18:22:22 -04:00
|
|
|
cq->nr_callbacks--;
|
2007-02-28 12:01:02 -05:00
|
|
|
if (!cq->nr_callbacks) {
|
2006-09-22 18:22:22 -04:00
|
|
|
list_del_init(cct->cq_list.next);
|
|
|
|
cct->cq_jobs--;
|
|
|
|
}
|
2007-02-15 11:07:30 -05:00
|
|
|
spin_unlock(&cq->task_lock);
|
2006-09-22 18:22:22 -04:00
|
|
|
}
|
|
|
|
|
2007-02-15 11:07:30 -05:00
|
|
|
spin_unlock_irqrestore(&cct->task_lock, flags);
|
2006-09-22 18:22:22 -04:00
|
|
|
}
|
|
|
|
|
|
|
|
static int comp_task(void *__cct)
|
|
|
|
{
|
2007-07-12 11:53:47 -04:00
|
|
|
struct ehca_cpu_comp_task *cct = __cct;
|
2007-02-15 11:07:30 -05:00
|
|
|
int cql_empty;
|
2006-09-22 18:22:22 -04:00
|
|
|
DECLARE_WAITQUEUE(wait, current);
|
|
|
|
|
|
|
|
set_current_state(TASK_INTERRUPTIBLE);
|
2007-07-12 11:53:47 -04:00
|
|
|
while (!kthread_should_stop()) {
|
2006-09-22 18:22:22 -04:00
|
|
|
add_wait_queue(&cct->wait_queue, &wait);
|
|
|
|
|
2007-02-15 11:07:30 -05:00
|
|
|
spin_lock_irq(&cct->task_lock);
|
|
|
|
cql_empty = list_empty(&cct->cq_list);
|
|
|
|
spin_unlock_irq(&cct->task_lock);
|
|
|
|
if (cql_empty)
|
2006-09-22 18:22:22 -04:00
|
|
|
schedule();
|
|
|
|
else
|
|
|
|
__set_current_state(TASK_RUNNING);
|
|
|
|
|
|
|
|
remove_wait_queue(&cct->wait_queue, &wait);
|
|
|
|
|
2007-02-15 11:07:30 -05:00
|
|
|
spin_lock_irq(&cct->task_lock);
|
|
|
|
cql_empty = list_empty(&cct->cq_list);
|
|
|
|
spin_unlock_irq(&cct->task_lock);
|
|
|
|
if (!cql_empty)
|
2006-09-22 18:22:22 -04:00
|
|
|
run_comp_task(__cct);
|
|
|
|
|
|
|
|
set_current_state(TASK_INTERRUPTIBLE);
|
|
|
|
}
|
|
|
|
__set_current_state(TASK_RUNNING);
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
static struct task_struct *create_comp_task(struct ehca_comp_pool *pool,
|
|
|
|
int cpu)
|
|
|
|
{
|
|
|
|
struct ehca_cpu_comp_task *cct;
|
|
|
|
|
|
|
|
cct = per_cpu_ptr(pool->cpu_comp_tasks, cpu);
|
|
|
|
spin_lock_init(&cct->task_lock);
|
|
|
|
INIT_LIST_HEAD(&cct->cq_list);
|
|
|
|
init_waitqueue_head(&cct->wait_queue);
|
|
|
|
cct->task = kthread_create(comp_task, cct, "ehca_comp/%d", cpu);
|
|
|
|
|
|
|
|
return cct->task;
|
|
|
|
}
|
|
|
|
|
|
|
|
static void destroy_comp_task(struct ehca_comp_pool *pool,
|
|
|
|
int cpu)
|
|
|
|
{
|
|
|
|
struct ehca_cpu_comp_task *cct;
|
|
|
|
struct task_struct *task;
|
|
|
|
unsigned long flags_cct;
|
|
|
|
|
|
|
|
cct = per_cpu_ptr(pool->cpu_comp_tasks, cpu);
|
|
|
|
|
|
|
|
spin_lock_irqsave(&cct->task_lock, flags_cct);
|
|
|
|
|
|
|
|
task = cct->task;
|
|
|
|
cct->task = NULL;
|
|
|
|
cct->cq_jobs = 0;
|
|
|
|
|
|
|
|
spin_unlock_irqrestore(&cct->task_lock, flags_cct);
|
|
|
|
|
|
|
|
if (task)
|
|
|
|
kthread_stop(task);
|
|
|
|
}
|
|
|
|
|
2007-08-22 19:28:30 -04:00
|
|
|
static void __cpuinit take_over_work(struct ehca_comp_pool *pool, int cpu)
|
2006-09-22 18:22:22 -04:00
|
|
|
{
|
|
|
|
struct ehca_cpu_comp_task *cct = per_cpu_ptr(pool->cpu_comp_tasks, cpu);
|
|
|
|
LIST_HEAD(list);
|
|
|
|
struct ehca_cq *cq;
|
|
|
|
unsigned long flags_cct;
|
|
|
|
|
|
|
|
spin_lock_irqsave(&cct->task_lock, flags_cct);
|
|
|
|
|
|
|
|
list_splice_init(&cct->cq_list, &list);
|
|
|
|
|
2007-07-12 11:53:47 -04:00
|
|
|
while (!list_empty(&list)) {
|
2007-02-15 11:06:33 -05:00
|
|
|
cq = list_entry(cct->cq_list.next, struct ehca_cq, entry);
|
2006-09-22 18:22:22 -04:00
|
|
|
|
2007-02-15 11:06:33 -05:00
|
|
|
list_del(&cq->entry);
|
2009-10-03 06:48:22 -04:00
|
|
|
__queue_comp_task(cq, this_cpu_ptr(pool->cpu_comp_tasks));
|
2006-09-22 18:22:22 -04:00
|
|
|
}
|
|
|
|
|
|
|
|
spin_unlock_irqrestore(&cct->task_lock, flags_cct);
|
|
|
|
|
|
|
|
}
|
|
|
|
|
2007-08-22 19:28:30 -04:00
|
|
|
static int __cpuinit comp_pool_callback(struct notifier_block *nfb,
|
|
|
|
unsigned long action,
|
|
|
|
void *hcpu)
|
2006-09-22 18:22:22 -04:00
|
|
|
{
|
|
|
|
unsigned int cpu = (unsigned long)hcpu;
|
|
|
|
struct ehca_cpu_comp_task *cct;
|
|
|
|
|
|
|
|
switch (action) {
|
|
|
|
case CPU_UP_PREPARE:
|
2007-05-09 05:35:10 -04:00
|
|
|
case CPU_UP_PREPARE_FROZEN:
|
2006-09-22 18:22:22 -04:00
|
|
|
ehca_gen_dbg("CPU: %x (CPU_PREPARE)", cpu);
|
2007-07-12 11:53:47 -04:00
|
|
|
if (!create_comp_task(pool, cpu)) {
|
2006-09-22 18:22:22 -04:00
|
|
|
ehca_gen_err("Can't create comp_task for cpu: %x", cpu);
|
2010-05-26 17:43:34 -04:00
|
|
|
return notifier_from_errno(-ENOMEM);
|
2006-09-22 18:22:22 -04:00
|
|
|
}
|
|
|
|
break;
|
|
|
|
case CPU_UP_CANCELED:
|
2007-05-09 05:35:10 -04:00
|
|
|
case CPU_UP_CANCELED_FROZEN:
|
2006-09-22 18:22:22 -04:00
|
|
|
ehca_gen_dbg("CPU: %x (CPU_CANCELED)", cpu);
|
|
|
|
cct = per_cpu_ptr(pool->cpu_comp_tasks, cpu);
|
2008-12-29 17:35:18 -05:00
|
|
|
kthread_bind(cct->task, cpumask_any(cpu_online_mask));
|
2006-09-22 18:22:22 -04:00
|
|
|
destroy_comp_task(pool, cpu);
|
|
|
|
break;
|
|
|
|
case CPU_ONLINE:
|
2007-05-09 05:35:10 -04:00
|
|
|
case CPU_ONLINE_FROZEN:
|
2006-09-22 18:22:22 -04:00
|
|
|
ehca_gen_dbg("CPU: %x (CPU_ONLINE)", cpu);
|
|
|
|
cct = per_cpu_ptr(pool->cpu_comp_tasks, cpu);
|
|
|
|
kthread_bind(cct->task, cpu);
|
|
|
|
wake_up_process(cct->task);
|
|
|
|
break;
|
|
|
|
case CPU_DOWN_PREPARE:
|
2007-05-09 05:35:10 -04:00
|
|
|
case CPU_DOWN_PREPARE_FROZEN:
|
2006-09-22 18:22:22 -04:00
|
|
|
ehca_gen_dbg("CPU: %x (CPU_DOWN_PREPARE)", cpu);
|
|
|
|
break;
|
|
|
|
case CPU_DOWN_FAILED:
|
2007-05-09 05:35:10 -04:00
|
|
|
case CPU_DOWN_FAILED_FROZEN:
|
2006-09-22 18:22:22 -04:00
|
|
|
ehca_gen_dbg("CPU: %x (CPU_DOWN_FAILED)", cpu);
|
|
|
|
break;
|
|
|
|
case CPU_DEAD:
|
2007-05-09 05:35:10 -04:00
|
|
|
case CPU_DEAD_FROZEN:
|
2006-09-22 18:22:22 -04:00
|
|
|
ehca_gen_dbg("CPU: %x (CPU_DEAD)", cpu);
|
|
|
|
destroy_comp_task(pool, cpu);
|
|
|
|
take_over_work(pool, cpu);
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
|
|
|
return NOTIFY_OK;
|
|
|
|
}
|
2007-08-22 19:28:30 -04:00
|
|
|
|
|
|
|
static struct notifier_block comp_pool_callback_nb __cpuinitdata = {
|
|
|
|
.notifier_call = comp_pool_callback,
|
|
|
|
.priority = 0,
|
|
|
|
};
|
2006-09-22 18:22:22 -04:00
|
|
|
|
|
|
|
int ehca_create_comp_pool(void)
|
|
|
|
{
|
|
|
|
int cpu;
|
|
|
|
struct task_struct *task;
|
|
|
|
|
2007-02-15 11:08:33 -05:00
|
|
|
if (!ehca_scaling_code)
|
|
|
|
return 0;
|
|
|
|
|
2006-09-22 18:22:22 -04:00
|
|
|
pool = kzalloc(sizeof(struct ehca_comp_pool), GFP_KERNEL);
|
|
|
|
if (pool == NULL)
|
|
|
|
return -ENOMEM;
|
|
|
|
|
|
|
|
spin_lock_init(&pool->last_cpu_lock);
|
2008-12-29 17:35:18 -05:00
|
|
|
pool->last_cpu = cpumask_any(cpu_online_mask);
|
2006-09-22 18:22:22 -04:00
|
|
|
|
|
|
|
pool->cpu_comp_tasks = alloc_percpu(struct ehca_cpu_comp_task);
|
|
|
|
if (pool->cpu_comp_tasks == NULL) {
|
|
|
|
kfree(pool);
|
|
|
|
return -EINVAL;
|
|
|
|
}
|
|
|
|
|
|
|
|
for_each_online_cpu(cpu) {
|
|
|
|
task = create_comp_task(pool, cpu);
|
|
|
|
if (task) {
|
|
|
|
kthread_bind(task, cpu);
|
|
|
|
wake_up_process(task);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2007-08-22 19:28:30 -04:00
|
|
|
register_hotcpu_notifier(&comp_pool_callback_nb);
|
2007-02-15 11:08:33 -05:00
|
|
|
|
|
|
|
printk(KERN_INFO "eHCA scaling code enabled\n");
|
2006-09-22 18:22:22 -04:00
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
void ehca_destroy_comp_pool(void)
|
|
|
|
{
|
|
|
|
int i;
|
|
|
|
|
2007-02-15 11:08:33 -05:00
|
|
|
if (!ehca_scaling_code)
|
|
|
|
return;
|
|
|
|
|
2007-08-22 19:28:30 -04:00
|
|
|
unregister_hotcpu_notifier(&comp_pool_callback_nb);
|
2006-09-22 18:22:22 -04:00
|
|
|
|
2008-12-29 17:35:17 -05:00
|
|
|
for_each_online_cpu(i)
|
|
|
|
destroy_comp_task(pool, i);
|
|
|
|
|
2007-02-05 19:21:09 -05:00
|
|
|
free_percpu(pool->cpu_comp_tasks);
|
|
|
|
kfree(pool);
|
2006-09-22 18:22:22 -04:00
|
|
|
}
|