From 5f77eeb05c9ee1de03fbe695b41362aae401726a Mon Sep 17 00:00:00 2001
From: Daniel Vetter <daniel.vetter@ffwll.ch>
Date: Mon, 8 Dec 2014 16:40:10 +0100
Subject: drm/i915: Use BUILD_BUG if possible in the i915 WARN_ON

Faster feedback to errors is always better. This is inspired by the
addition to WARN_ONs to mask/enable helpers for registers to make sure
callers have the arguments ordered correctly: Pretty much always the
arguments are static.

We use WARN_ON(1) a lot in default switch statements though where we
should always handle all cases. So add a new macro specifically for
that.

The idea to use __builtin_constant_p is from Chris Wilson.

v2: Use the ({}) gcc-ism to avoid the static inline, suggested by
Dave. My first attempt used __cond as the temp var, which is the same
used by BUILD_BUG_ON, but with inverted sense. Hilarity ensued, so
sprinkle i915 into the name.

Also use a temporary variable to only evaluate the condition once,
suggested by Damien.

v3: It's crazy but apparently 32bit gcc can't compile out the
BUILD_BUG_ON in a lot of cases and just falls over. I have no idea
why, but until clue grows just disable this nifty idea on 32bit
builds. Reported by 0-day builder.

v4: Got it all wrong, apparently its the gcc version. We need 4.9+.
Now reported by Imre.

v5: Chris suggested to add the case to MISSING_CASE for speedier
debug.

v6: Even some gcc 4.9 versions don't see through the maze, so give up
for now. Keep the skeleton and MISSING_CASE stuff though.

Cc: Imre Deak <imre.deak@intel.com>
Cc: Damien Lespiau <damien.lespiau@intel.com>
Cc: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Jani Nikula <jani.nikula@linux.intel.com>
Cc: Dave Gordon <david.s.gordon@intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@intel.com>
---
 drivers/gpu/drm/i915/intel_uncore.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

(limited to 'drivers/gpu/drm/i915/intel_uncore.c')

diff --git a/drivers/gpu/drm/i915/intel_uncore.c b/drivers/gpu/drm/i915/intel_uncore.c
index 46de8d75b4bf..883d2febb705 100644
--- a/drivers/gpu/drm/i915/intel_uncore.c
+++ b/drivers/gpu/drm/i915/intel_uncore.c
@@ -1202,7 +1202,7 @@ void intel_uncore_init(struct drm_device *dev)
 
 	switch (INTEL_INFO(dev)->gen) {
 	default:
-		WARN_ON(1);
+		MISSING_CASE(INTEL_INFO(dev)->gen);
 		return;
 	case 9:
 		ASSIGN_WRITE_MMIO_VFUNCS(gen9);
@@ -1300,7 +1300,7 @@ int i915_reg_read_ioctl(struct drm_device *dev,
 		reg->val = I915_READ8(reg->offset);
 		break;
 	default:
-		WARN_ON(1);
+		MISSING_CASE(entry->size);
 		ret = -EINVAL;
 		goto out;
 	}
-- 
cgit v1.2.3


From db5ff4ac97f6602360645414b698a05f91b40542 Mon Sep 17 00:00:00 2001
From: Deepak S <deepak.s@linux.intel.com>
Date: Thu, 11 Dec 2014 21:42:49 +0530
Subject: drm/i915: Forcewake Register Range changes for CHV
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

According to updated BSpec, Render/Common/media Wells register range changed.
Updating the same to match the spec and avoid extra forcewake for none
forcewake range.

v2: Update media forcewake range (Ville)

Signed-off-by: Deepak S <deepak.s@linux.intel.com>
Reviewed-by: Ville Syrjälä <ville.syrjala@linux.intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
---
 drivers/gpu/drm/i915/intel_uncore.c | 11 ++++-------
 1 file changed, 4 insertions(+), 7 deletions(-)

(limited to 'drivers/gpu/drm/i915/intel_uncore.c')

diff --git a/drivers/gpu/drm/i915/intel_uncore.c b/drivers/gpu/drm/i915/intel_uncore.c
index 883d2febb705..e9561de382aa 100644
--- a/drivers/gpu/drm/i915/intel_uncore.c
+++ b/drivers/gpu/drm/i915/intel_uncore.c
@@ -647,9 +647,9 @@ void assert_force_wake_inactive(struct drm_i915_private *dev_priv)
 
 #define FORCEWAKE_CHV_RENDER_RANGE_OFFSET(reg) \
 	(REG_RANGE((reg), 0x2000, 0x4000) || \
-	 REG_RANGE((reg), 0x5000, 0x8000) || \
+	 REG_RANGE((reg), 0x5200, 0x8000) || \
 	 REG_RANGE((reg), 0x8300, 0x8500) || \
-	 REG_RANGE((reg), 0xB000, 0xC000) || \
+	 REG_RANGE((reg), 0xB000, 0xB480) || \
 	 REG_RANGE((reg), 0xE000, 0xE800))
 
 #define FORCEWAKE_CHV_MEDIA_RANGE_OFFSET(reg) \
@@ -658,17 +658,14 @@ void assert_force_wake_inactive(struct drm_i915_private *dev_priv)
 	 REG_RANGE((reg), 0x12000, 0x14000) || \
 	 REG_RANGE((reg), 0x1A000, 0x1C000) || \
 	 REG_RANGE((reg), 0x1E800, 0x1EA00) || \
-	 REG_RANGE((reg), 0x30000, 0x40000))
+	 REG_RANGE((reg), 0x30000, 0x38000))
 
 #define FORCEWAKE_CHV_COMMON_RANGE_OFFSET(reg) \
 	(REG_RANGE((reg), 0x4000, 0x5000) || \
 	 REG_RANGE((reg), 0x8000, 0x8300) || \
 	 REG_RANGE((reg), 0x8500, 0x8600) || \
 	 REG_RANGE((reg), 0x9000, 0xB000) || \
-	 REG_RANGE((reg), 0xC000, 0xC800) || \
-	 REG_RANGE((reg), 0xF000, 0x10000) || \
-	 REG_RANGE((reg), 0x14000, 0x14400) || \
-	 REG_RANGE((reg), 0x22000, 0x24000))
+	 REG_RANGE((reg), 0xF000, 0x10000))
 
 #define FORCEWAKE_GEN9_UNCORE_RANGE_OFFSET(reg) \
 	REG_RANGE((reg), 0xB00,  0x2000)
-- 
cgit v1.2.3


From dc9fb09cae5b1355c1a9d8102e40b97b34332f31 Mon Sep 17 00:00:00 2001
From: Chris Wilson <chris@chris-wilson.co.uk>
Date: Fri, 16 Jan 2015 11:34:34 +0200
Subject: drm/i915: Rebalance runtime pm vs forcewake

Calling intel_runtime_pm_put() is illegal from a soft-irq context, so
revert the crude hack

commit aa0b3b5bb8768c1a6a6788869d9c7015eae7e80c
Author: Paulo Zanoni <paulo.r.zanoni@intel.com>
Date:   Tue Apr 1 14:55:07 2014 -0300

    drm/i915: don't schedule force_wake_timer at gen6_read

and apply the single line corrective instead.

v2: assert forcewake is off after the forcewake_reset (Paulo)

References: https://bugs.freedesktop.org/show_bug.cgi?id=80913
Cc: Paulo Zanoni <paulo.r.zanoni@intel.com>
Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: Mika Kuoppala <mika.kuoppala@intel.com>
Reviewed-by: Deepak S <deepak.s@linux.intel.com> (v1)
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
---
 drivers/gpu/drm/i915/i915_drv.c     |  5 +++--
 drivers/gpu/drm/i915/intel_uncore.c | 18 ++++++------------
 2 files changed, 9 insertions(+), 14 deletions(-)

(limited to 'drivers/gpu/drm/i915/intel_uncore.c')

diff --git a/drivers/gpu/drm/i915/i915_drv.c b/drivers/gpu/drm/i915/i915_drv.c
index 308774f42079..55a3fef075ae 100644
--- a/drivers/gpu/drm/i915/i915_drv.c
+++ b/drivers/gpu/drm/i915/i915_drv.c
@@ -1365,8 +1365,6 @@ static int intel_runtime_suspend(struct device *device)
 	if (WARN_ON_ONCE(!HAS_RUNTIME_PM(dev)))
 		return -ENODEV;
 
-	assert_force_wake_inactive(dev_priv);
-
 	DRM_DEBUG_KMS("Suspending device\n");
 
 	/*
@@ -1405,6 +1403,7 @@ static int intel_runtime_suspend(struct device *device)
 	}
 
 	del_timer_sync(&dev_priv->gpu_error.hangcheck_timer);
+	intel_uncore_forcewake_reset(dev, false);
 	dev_priv->pm.suspended = true;
 
 	/*
@@ -1432,6 +1431,8 @@ static int intel_runtime_suspend(struct device *device)
 		intel_opregion_notify_adapter(dev, PCI_D3hot);
 	}
 
+	assert_force_wake_inactive(dev_priv);
+
 	DRM_DEBUG_KMS("Device suspended\n");
 	return 0;
 }
diff --git a/drivers/gpu/drm/i915/intel_uncore.c b/drivers/gpu/drm/i915/intel_uncore.c
index e9561de382aa..b39ed7968383 100644
--- a/drivers/gpu/drm/i915/intel_uncore.c
+++ b/drivers/gpu/drm/i915/intel_uncore.c
@@ -449,8 +449,6 @@ static void gen6_force_wake_timer(unsigned long arg)
 	if (--dev_priv->uncore.forcewake_count == 0)
 		dev_priv->uncore.funcs.force_wake_put(dev_priv, FORCEWAKE_ALL);
 	spin_unlock_irqrestore(&dev_priv->uncore.lock, irqflags);
-
-	intel_runtime_pm_put(dev_priv);
 }
 
 void intel_uncore_forcewake_reset(struct drm_device *dev, bool restore)
@@ -586,7 +584,6 @@ void gen6_gt_force_wake_get(struct drm_i915_private *dev_priv, int fw_engine)
 void gen6_gt_force_wake_put(struct drm_i915_private *dev_priv, int fw_engine)
 {
 	unsigned long irqflags;
-	bool delayed = false;
 
 	if (!dev_priv->uncore.funcs.force_wake_put)
 		return;
@@ -603,21 +600,19 @@ void gen6_gt_force_wake_put(struct drm_i915_private *dev_priv, int fw_engine)
 		goto out;
 	}
 
-
 	spin_lock_irqsave(&dev_priv->uncore.lock, irqflags);
 	WARN_ON(!dev_priv->uncore.forcewake_count);
 
 	if (--dev_priv->uncore.forcewake_count == 0) {
 		dev_priv->uncore.forcewake_count++;
-		delayed = true;
 		mod_timer_pinned(&dev_priv->uncore.force_wake_timer,
 				 jiffies + 1);
 	}
+
 	spin_unlock_irqrestore(&dev_priv->uncore.lock, irqflags);
 
 out:
-	if (!delayed)
-		intel_runtime_pm_put(dev_priv);
+	intel_runtime_pm_put(dev_priv);
 }
 
 void assert_force_wake_inactive(struct drm_i915_private *dev_priv)
@@ -774,12 +769,11 @@ gen6_read##x(struct drm_i915_private *dev_priv, off_t reg, bool trace) { \
 	    NEEDS_FORCE_WAKE((dev_priv), (reg))) { \
 		dev_priv->uncore.funcs.force_wake_get(dev_priv, \
 						      FORCEWAKE_ALL); \
-		val = __raw_i915_read##x(dev_priv, reg); \
-		dev_priv->uncore.funcs.force_wake_put(dev_priv, \
-						      FORCEWAKE_ALL); \
-	} else { \
-		val = __raw_i915_read##x(dev_priv, reg); \
+		dev_priv->uncore.forcewake_count++; \
+		mod_timer_pinned(&dev_priv->uncore.force_wake_timer, \
+				 jiffies + 1); \
 	} \
+	val = __raw_i915_read##x(dev_priv, reg); \
 	hsw_unclaimed_reg_debug(dev_priv, reg, true, false); \
 	REG_READ_FOOTER; \
 }
-- 
cgit v1.2.3


From 6daccb0b2a4c8c675395061cffbf77e9e5f31e7b Mon Sep 17 00:00:00 2001
From: Chris Wilson <chris@chris-wilson.co.uk>
Date: Fri, 16 Jan 2015 11:34:35 +0200
Subject: drm/i915: Assert that runtime pm is active on user fw access

On user forcewake access, assert that runtime pm reference is held.
Fix and cleanup the callsites accordingly.

v2: Remove intel_runtime_pm_get() rebasehap (Deepak)

v3: use drivers own runtime state tracking as pm_runtime_active()
    will return wrong results when we are in resume callchain (Mika)

Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: Mika Kuoppala <mika.kuoppala@intel.com>
Reviewed-by: Deepak S <deepak.s@linux.intel.com> (v2)
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
---
 drivers/gpu/drm/i915/i915_debugfs.c  |  2 +
 drivers/gpu/drm/i915/intel_display.c | 19 +--------
 drivers/gpu/drm/i915/intel_lrc.c     | 53 ++-----------------------
 drivers/gpu/drm/i915/intel_uncore.c  | 76 +++++++++++-------------------------
 4 files changed, 30 insertions(+), 120 deletions(-)

(limited to 'drivers/gpu/drm/i915/intel_uncore.c')

diff --git a/drivers/gpu/drm/i915/i915_debugfs.c b/drivers/gpu/drm/i915/i915_debugfs.c
index 7952f313dda2..ac517eb07c00 100644
--- a/drivers/gpu/drm/i915/i915_debugfs.c
+++ b/drivers/gpu/drm/i915/i915_debugfs.c
@@ -4339,6 +4339,7 @@ static int i915_forcewake_open(struct inode *inode, struct file *file)
 	if (INTEL_INFO(dev)->gen < 6)
 		return 0;
 
+	intel_runtime_pm_get(dev_priv);
 	gen6_gt_force_wake_get(dev_priv, FORCEWAKE_ALL);
 
 	return 0;
@@ -4353,6 +4354,7 @@ static int i915_forcewake_release(struct inode *inode, struct file *file)
 		return 0;
 
 	gen6_gt_force_wake_put(dev_priv, FORCEWAKE_ALL);
+	intel_runtime_pm_put(dev_priv);
 
 	return 0;
 }
diff --git a/drivers/gpu/drm/i915/intel_display.c b/drivers/gpu/drm/i915/intel_display.c
index c5cbcd74387a..61b123f1a652 100644
--- a/drivers/gpu/drm/i915/intel_display.c
+++ b/drivers/gpu/drm/i915/intel_display.c
@@ -7870,19 +7870,8 @@ static void hsw_restore_lcpll(struct drm_i915_private *dev_priv)
 	/*
 	 * Make sure we're not on PC8 state before disabling PC8, otherwise
 	 * we'll hang the machine. To prevent PC8 state, just enable force_wake.
-	 *
-	 * The other problem is that hsw_restore_lcpll() is called as part of
-	 * the runtime PM resume sequence, so we can't just call
-	 * gen6_gt_force_wake_get() because that function calls
-	 * intel_runtime_pm_get(), and we can't change the runtime PM refcount
-	 * while we are on the resume sequence. So to solve this problem we have
-	 * to call special forcewake code that doesn't touch runtime PM and
-	 * doesn't enable the forcewake delayed work.
 	 */
-	spin_lock_irq(&dev_priv->uncore.lock);
-	if (dev_priv->uncore.forcewake_count++ == 0)
-		dev_priv->uncore.funcs.force_wake_get(dev_priv, FORCEWAKE_ALL);
-	spin_unlock_irq(&dev_priv->uncore.lock);
+	gen6_gt_force_wake_get(dev_priv, FORCEWAKE_ALL);
 
 	if (val & LCPLL_POWER_DOWN_ALLOW) {
 		val &= ~LCPLL_POWER_DOWN_ALLOW;
@@ -7912,11 +7901,7 @@ static void hsw_restore_lcpll(struct drm_i915_private *dev_priv)
 			DRM_ERROR("Switching back to LCPLL failed\n");
 	}
 
-	/* See the big comment above. */
-	spin_lock_irq(&dev_priv->uncore.lock);
-	if (--dev_priv->uncore.forcewake_count == 0)
-		dev_priv->uncore.funcs.force_wake_put(dev_priv, FORCEWAKE_ALL);
-	spin_unlock_irq(&dev_priv->uncore.lock);
+	gen6_gt_force_wake_put(dev_priv, FORCEWAKE_ALL);
 }
 
 /*
diff --git a/drivers/gpu/drm/i915/intel_lrc.c b/drivers/gpu/drm/i915/intel_lrc.c
index a5f614de0828..49c7d862f8a5 100644
--- a/drivers/gpu/drm/i915/intel_lrc.c
+++ b/drivers/gpu/drm/i915/intel_lrc.c
@@ -283,7 +283,6 @@ static void execlists_elsp_write(struct intel_engine_cs *ring,
 	struct drm_i915_private *dev_priv = dev->dev_private;
 	uint64_t temp = 0;
 	uint32_t desc[4];
-	unsigned long flags;
 
 	/* XXX: You must always write both descriptors in the order below. */
 	if (ctx_obj1)
@@ -297,63 +296,17 @@ static void execlists_elsp_write(struct intel_engine_cs *ring,
 	desc[3] = (u32)(temp >> 32);
 	desc[2] = (u32)temp;
 
-	/* Set Force Wakeup bit to prevent GT from entering C6 while ELSP writes
-	 * are in progress.
-	 *
-	 * The other problem is that we can't just call gen6_gt_force_wake_get()
-	 * because that function calls intel_runtime_pm_get(), which might sleep.
-	 * Instead, we do the runtime_pm_get/put when creating/destroying requests.
-	 */
-	spin_lock_irqsave(&dev_priv->uncore.lock, flags);
-	if (IS_CHERRYVIEW(dev) || INTEL_INFO(dev)->gen >= 9) {
-		if (dev_priv->uncore.fw_rendercount++ == 0)
-			dev_priv->uncore.funcs.force_wake_get(dev_priv,
-							      FORCEWAKE_RENDER);
-		if (dev_priv->uncore.fw_mediacount++ == 0)
-			dev_priv->uncore.funcs.force_wake_get(dev_priv,
-							      FORCEWAKE_MEDIA);
-		if (INTEL_INFO(dev)->gen >= 9) {
-			if (dev_priv->uncore.fw_blittercount++ == 0)
-				dev_priv->uncore.funcs.force_wake_get(dev_priv,
-							FORCEWAKE_BLITTER);
-		}
-	} else {
-		if (dev_priv->uncore.forcewake_count++ == 0)
-			dev_priv->uncore.funcs.force_wake_get(dev_priv,
-							      FORCEWAKE_ALL);
-	}
-	spin_unlock_irqrestore(&dev_priv->uncore.lock, flags);
-
+	gen6_gt_force_wake_get(dev_priv, FORCEWAKE_ALL);
 	I915_WRITE(RING_ELSP(ring), desc[1]);
 	I915_WRITE(RING_ELSP(ring), desc[0]);
 	I915_WRITE(RING_ELSP(ring), desc[3]);
+
 	/* The context is automatically loaded after the following */
 	I915_WRITE(RING_ELSP(ring), desc[2]);
 
 	/* ELSP is a wo register, so use another nearby reg for posting instead */
 	POSTING_READ(RING_EXECLIST_STATUS(ring));
-
-	/* Release Force Wakeup (see the big comment above). */
-	spin_lock_irqsave(&dev_priv->uncore.lock, flags);
-	if (IS_CHERRYVIEW(dev) || INTEL_INFO(dev)->gen >= 9) {
-		if (--dev_priv->uncore.fw_rendercount == 0)
-			dev_priv->uncore.funcs.force_wake_put(dev_priv,
-							      FORCEWAKE_RENDER);
-		if (--dev_priv->uncore.fw_mediacount == 0)
-			dev_priv->uncore.funcs.force_wake_put(dev_priv,
-							      FORCEWAKE_MEDIA);
-		if (INTEL_INFO(dev)->gen >= 9) {
-			if (--dev_priv->uncore.fw_blittercount == 0)
-				dev_priv->uncore.funcs.force_wake_put(dev_priv,
-							FORCEWAKE_BLITTER);
-		}
-	} else {
-		if (--dev_priv->uncore.forcewake_count == 0)
-			dev_priv->uncore.funcs.force_wake_put(dev_priv,
-							      FORCEWAKE_ALL);
-	}
-
-	spin_unlock_irqrestore(&dev_priv->uncore.lock, flags);
+	gen6_gt_force_wake_put(dev_priv, FORCEWAKE_ALL);
 }
 
 static int execlists_update_context(struct drm_i915_gem_object *ctx_obj,
diff --git a/drivers/gpu/drm/i915/intel_uncore.c b/drivers/gpu/drm/i915/intel_uncore.c
index b39ed7968383..974881e23ffa 100644
--- a/drivers/gpu/drm/i915/intel_uncore.c
+++ b/drivers/gpu/drm/i915/intel_uncore.c
@@ -24,6 +24,8 @@
 #include "i915_drv.h"
 #include "intel_drv.h"
 
+#include <linux/pm_runtime.h>
+
 #define FORCEWAKE_ACK_TIMEOUT_MS 2
 
 #define __raw_i915_read8(dev_priv__, reg__) readb((dev_priv__)->regs + (reg__))
@@ -247,10 +249,6 @@ static void __vlv_force_wake_put(struct drm_i915_private *dev_priv,
 
 static void vlv_force_wake_get(struct drm_i915_private *dev_priv, int fw_engine)
 {
-	unsigned long irqflags;
-
-	spin_lock_irqsave(&dev_priv->uncore.lock, irqflags);
-
 	if (fw_engine & FORCEWAKE_RENDER &&
 	    dev_priv->uncore.fw_rendercount++ != 0)
 		fw_engine &= ~FORCEWAKE_RENDER;
@@ -260,16 +258,10 @@ static void vlv_force_wake_get(struct drm_i915_private *dev_priv, int fw_engine)
 
 	if (fw_engine)
 		dev_priv->uncore.funcs.force_wake_get(dev_priv, fw_engine);
-
-	spin_unlock_irqrestore(&dev_priv->uncore.lock, irqflags);
 }
 
 static void vlv_force_wake_put(struct drm_i915_private *dev_priv, int fw_engine)
 {
-	unsigned long irqflags;
-
-	spin_lock_irqsave(&dev_priv->uncore.lock, irqflags);
-
 	if (fw_engine & FORCEWAKE_RENDER) {
 		WARN_ON(!dev_priv->uncore.fw_rendercount);
 		if (--dev_priv->uncore.fw_rendercount != 0)
@@ -284,8 +276,6 @@ static void vlv_force_wake_put(struct drm_i915_private *dev_priv, int fw_engine)
 
 	if (fw_engine)
 		dev_priv->uncore.funcs.force_wake_put(dev_priv, fw_engine);
-
-	spin_unlock_irqrestore(&dev_priv->uncore.lock, irqflags);
 }
 
 static void __gen9_gt_force_wake_mt_reset(struct drm_i915_private *dev_priv)
@@ -380,10 +370,6 @@ __gen9_force_wake_put(struct drm_i915_private *dev_priv, int fw_engine)
 static void
 gen9_force_wake_get(struct drm_i915_private *dev_priv, int fw_engine)
 {
-	unsigned long irqflags;
-
-	spin_lock_irqsave(&dev_priv->uncore.lock, irqflags);
-
 	if (FORCEWAKE_RENDER & fw_engine) {
 		if (dev_priv->uncore.fw_rendercount++ == 0)
 			dev_priv->uncore.funcs.force_wake_get(dev_priv,
@@ -401,17 +387,11 @@ gen9_force_wake_get(struct drm_i915_private *dev_priv, int fw_engine)
 			dev_priv->uncore.funcs.force_wake_get(dev_priv,
 							FORCEWAKE_BLITTER);
 	}
-
-	spin_unlock_irqrestore(&dev_priv->uncore.lock, irqflags);
 }
 
 static void
 gen9_force_wake_put(struct drm_i915_private *dev_priv, int fw_engine)
 {
-	unsigned long irqflags;
-
-	spin_lock_irqsave(&dev_priv->uncore.lock, irqflags);
-
 	if (FORCEWAKE_RENDER & fw_engine) {
 		WARN_ON(dev_priv->uncore.fw_rendercount == 0);
 		if (--dev_priv->uncore.fw_rendercount == 0)
@@ -432,8 +412,6 @@ gen9_force_wake_put(struct drm_i915_private *dev_priv, int fw_engine)
 			dev_priv->uncore.funcs.force_wake_put(dev_priv,
 							FORCEWAKE_BLITTER);
 	}
-
-	spin_unlock_irqrestore(&dev_priv->uncore.lock, irqflags);
 }
 
 static void gen6_force_wake_timer(unsigned long arg)
@@ -562,19 +540,20 @@ void gen6_gt_force_wake_get(struct drm_i915_private *dev_priv, int fw_engine)
 	if (!dev_priv->uncore.funcs.force_wake_get)
 		return;
 
-	intel_runtime_pm_get(dev_priv);
+	WARN_ON(dev_priv->pm.suspended);
 
-	/* Redirect to Gen9 specific routine */
-	if (IS_GEN9(dev_priv->dev))
-		return gen9_force_wake_get(dev_priv, fw_engine);
+	spin_lock_irqsave(&dev_priv->uncore.lock, irqflags);
 
-	/* Redirect to VLV specific routine */
-	if (IS_VALLEYVIEW(dev_priv->dev))
-		return vlv_force_wake_get(dev_priv, fw_engine);
+	if (IS_GEN9(dev_priv->dev)) {
+		gen9_force_wake_get(dev_priv, fw_engine);
+	} else if (IS_VALLEYVIEW(dev_priv->dev)) {
+		vlv_force_wake_get(dev_priv, fw_engine);
+	} else {
+		if (dev_priv->uncore.forcewake_count++ == 0)
+			dev_priv->uncore.funcs.force_wake_get(dev_priv,
+							      FORCEWAKE_ALL);
+	}
 
-	spin_lock_irqsave(&dev_priv->uncore.lock, irqflags);
-	if (dev_priv->uncore.forcewake_count++ == 0)
-		dev_priv->uncore.funcs.force_wake_get(dev_priv, FORCEWAKE_ALL);
 	spin_unlock_irqrestore(&dev_priv->uncore.lock, irqflags);
 }
 
@@ -588,31 +567,22 @@ void gen6_gt_force_wake_put(struct drm_i915_private *dev_priv, int fw_engine)
 	if (!dev_priv->uncore.funcs.force_wake_put)
 		return;
 
-	/* Redirect to Gen9 specific routine */
+	spin_lock_irqsave(&dev_priv->uncore.lock, irqflags);
+
 	if (IS_GEN9(dev_priv->dev)) {
 		gen9_force_wake_put(dev_priv, fw_engine);
-		goto out;
-	}
-
-	/* Redirect to VLV specific routine */
-	if (IS_VALLEYVIEW(dev_priv->dev)) {
+	} else if (IS_VALLEYVIEW(dev_priv->dev)) {
 		vlv_force_wake_put(dev_priv, fw_engine);
-		goto out;
-	}
-
-	spin_lock_irqsave(&dev_priv->uncore.lock, irqflags);
-	WARN_ON(!dev_priv->uncore.forcewake_count);
-
-	if (--dev_priv->uncore.forcewake_count == 0) {
-		dev_priv->uncore.forcewake_count++;
-		mod_timer_pinned(&dev_priv->uncore.force_wake_timer,
-				 jiffies + 1);
+	} else {
+		WARN_ON(!dev_priv->uncore.forcewake_count);
+		if (--dev_priv->uncore.forcewake_count == 0) {
+			dev_priv->uncore.forcewake_count++;
+			mod_timer_pinned(&dev_priv->uncore.force_wake_timer,
+					 jiffies + 1);
+		}
 	}
 
 	spin_unlock_irqrestore(&dev_priv->uncore.lock, irqflags);
-
-out:
-	intel_runtime_pm_put(dev_priv);
 }
 
 void assert_force_wake_inactive(struct drm_i915_private *dev_priv)
-- 
cgit v1.2.3


From 51f67885842e36e0d0c853795cd02b129e19a20b Mon Sep 17 00:00:00 2001
From: Chris Wilson <chris@chris-wilson.co.uk>
Date: Fri, 16 Jan 2015 11:34:36 +0200
Subject: drm/i915: Skip uncore lock on earlier gens

With gen < 6 we don't need to take uncore lock as we
don't have anything to protect from concurrent access.

v2: rebase and account for gen9 changes

Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> (v1)
Signed-off-by: Mika Kuoppala <mika.kuoppala@intel.com>
Reviewed-by: Deepak S <deepak.s@linux.intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
---
 drivers/gpu/drm/i915/intel_uncore.c | 158 +++++++++++++++++++++---------------
 1 file changed, 91 insertions(+), 67 deletions(-)

(limited to 'drivers/gpu/drm/i915/intel_uncore.c')

diff --git a/drivers/gpu/drm/i915/intel_uncore.c b/drivers/gpu/drm/i915/intel_uncore.c
index 974881e23ffa..3d1ffacb7612 100644
--- a/drivers/gpu/drm/i915/intel_uncore.c
+++ b/drivers/gpu/drm/i915/intel_uncore.c
@@ -702,38 +702,61 @@ hsw_unclaimed_reg_detect(struct drm_i915_private *dev_priv)
 	}
 }
 
-#define REG_READ_HEADER(x) \
-	unsigned long irqflags; \
+#define GEN2_READ_HEADER(x) \
 	u##x val = 0; \
-	assert_device_not_suspended(dev_priv); \
-	spin_lock_irqsave(&dev_priv->uncore.lock, irqflags)
+	assert_device_not_suspended(dev_priv);
 
-#define REG_READ_FOOTER \
-	spin_unlock_irqrestore(&dev_priv->uncore.lock, irqflags); \
+#define GEN2_READ_FOOTER \
 	trace_i915_reg_rw(false, reg, val, sizeof(val), trace); \
 	return val
 
-#define __gen4_read(x) \
+#define __gen2_read(x) \
 static u##x \
-gen4_read##x(struct drm_i915_private *dev_priv, off_t reg, bool trace) { \
-	REG_READ_HEADER(x); \
+gen2_read##x(struct drm_i915_private *dev_priv, off_t reg, bool trace) { \
+	GEN2_READ_HEADER(x); \
 	val = __raw_i915_read##x(dev_priv, reg); \
-	REG_READ_FOOTER; \
+	GEN2_READ_FOOTER; \
 }
 
 #define __gen5_read(x) \
 static u##x \
 gen5_read##x(struct drm_i915_private *dev_priv, off_t reg, bool trace) { \
-	REG_READ_HEADER(x); \
+	GEN2_READ_HEADER(x); \
 	ilk_dummy_write(dev_priv); \
 	val = __raw_i915_read##x(dev_priv, reg); \
-	REG_READ_FOOTER; \
+	GEN2_READ_FOOTER; \
 }
 
+__gen5_read(8)
+__gen5_read(16)
+__gen5_read(32)
+__gen5_read(64)
+__gen2_read(8)
+__gen2_read(16)
+__gen2_read(32)
+__gen2_read(64)
+
+#undef __gen5_read
+#undef __gen2_read
+
+#undef GEN2_READ_FOOTER
+#undef GEN2_READ_HEADER
+
+#define GEN6_READ_HEADER(x) \
+	unsigned long irqflags; \
+	u##x val = 0; \
+	assert_device_not_suspended(dev_priv); \
+	spin_lock_irqsave(&dev_priv->uncore.lock, irqflags)
+
+#define GEN6_READ_FOOTER \
+	spin_unlock_irqrestore(&dev_priv->uncore.lock, irqflags); \
+	trace_i915_reg_rw(false, reg, val, sizeof(val), trace); \
+	return val
+
 #define __gen6_read(x) \
 static u##x \
 gen6_read##x(struct drm_i915_private *dev_priv, off_t reg, bool trace) { \
-	REG_READ_HEADER(x); \
+	GEN6_READ_HEADER(x); \
 	hsw_unclaimed_reg_debug(dev_priv, reg, true, true); \
 	if (dev_priv->uncore.forcewake_count == 0 && \
 	    NEEDS_FORCE_WAKE((dev_priv), (reg))) { \
@@ -745,14 +768,14 @@ gen6_read##x(struct drm_i915_private *dev_priv, off_t reg, bool trace) { \
 	} \
 	val = __raw_i915_read##x(dev_priv, reg); \
 	hsw_unclaimed_reg_debug(dev_priv, reg, true, false); \
-	REG_READ_FOOTER; \
+	GEN6_READ_FOOTER; \
 }
 
 #define __vlv_read(x) \
 static u##x \
 vlv_read##x(struct drm_i915_private *dev_priv, off_t reg, bool trace) { \
 	unsigned fwengine = 0; \
-	REG_READ_HEADER(x); \
+	GEN6_READ_HEADER(x); \
 	if (FORCEWAKE_VLV_RENDER_RANGE_OFFSET(reg)) { \
 		if (dev_priv->uncore.fw_rendercount == 0) \
 			fwengine = FORCEWAKE_RENDER; \
@@ -765,14 +788,14 @@ vlv_read##x(struct drm_i915_private *dev_priv, off_t reg, bool trace) { \
 	val = __raw_i915_read##x(dev_priv, reg); \
 	if (fwengine) \
 		dev_priv->uncore.funcs.force_wake_put(dev_priv, fwengine); \
-	REG_READ_FOOTER; \
+	GEN6_READ_FOOTER; \
 }
 
 #define __chv_read(x) \
 static u##x \
 chv_read##x(struct drm_i915_private *dev_priv, off_t reg, bool trace) { \
 	unsigned fwengine = 0; \
-	REG_READ_HEADER(x); \
+	GEN6_READ_HEADER(x); \
 	if (FORCEWAKE_CHV_RENDER_RANGE_OFFSET(reg)) { \
 		if (dev_priv->uncore.fw_rendercount == 0) \
 			fwengine = FORCEWAKE_RENDER; \
@@ -790,7 +813,7 @@ chv_read##x(struct drm_i915_private *dev_priv, off_t reg, bool trace) { \
 	val = __raw_i915_read##x(dev_priv, reg); \
 	if (fwengine) \
 		dev_priv->uncore.funcs.force_wake_put(dev_priv, fwengine); \
-	REG_READ_FOOTER; \
+	GEN6_READ_FOOTER; \
 }
 
 #define SKL_NEEDS_FORCE_WAKE(dev_priv, reg)	\
@@ -799,7 +822,7 @@ chv_read##x(struct drm_i915_private *dev_priv, off_t reg, bool trace) { \
 #define __gen9_read(x) \
 static u##x \
 gen9_read##x(struct drm_i915_private *dev_priv, off_t reg, bool trace) { \
-	REG_READ_HEADER(x); \
+	GEN6_READ_HEADER(x); \
 	if (!SKL_NEEDS_FORCE_WAKE((dev_priv), (reg))) { \
 		val = __raw_i915_read##x(dev_priv, reg); \
 	} else { \
@@ -825,7 +848,7 @@ gen9_read##x(struct drm_i915_private *dev_priv, off_t reg, bool trace) { \
 		if (fwengine) \
 			dev_priv->uncore.funcs.force_wake_put(dev_priv, fwengine); \
 	} \
-	REG_READ_FOOTER; \
+	GEN6_READ_FOOTER; \
 }
 
 __gen9_read(8)
@@ -844,55 +867,66 @@ __gen6_read(8)
 __gen6_read(16)
 __gen6_read(32)
 __gen6_read(64)
-__gen5_read(8)
-__gen5_read(16)
-__gen5_read(32)
-__gen5_read(64)
-__gen4_read(8)
-__gen4_read(16)
-__gen4_read(32)
-__gen4_read(64)
 
 #undef __gen9_read
 #undef __chv_read
 #undef __vlv_read
 #undef __gen6_read
-#undef __gen5_read
-#undef __gen4_read
-#undef REG_READ_FOOTER
-#undef REG_READ_HEADER
+#undef GEN6_READ_FOOTER
+#undef GEN6_READ_HEADER
 
-#define REG_WRITE_HEADER \
-	unsigned long irqflags; \
+#define GEN2_WRITE_HEADER \
 	trace_i915_reg_rw(true, reg, val, sizeof(val), trace); \
 	assert_device_not_suspended(dev_priv); \
-	spin_lock_irqsave(&dev_priv->uncore.lock, irqflags)
 
-#define REG_WRITE_FOOTER \
-	spin_unlock_irqrestore(&dev_priv->uncore.lock, irqflags)
+#define GEN2_WRITE_FOOTER
 
-#define __gen4_write(x) \
+#define __gen2_write(x) \
 static void \
-gen4_write##x(struct drm_i915_private *dev_priv, off_t reg, u##x val, bool trace) { \
-	REG_WRITE_HEADER; \
+gen2_write##x(struct drm_i915_private *dev_priv, off_t reg, u##x val, bool trace) { \
+	GEN2_WRITE_HEADER; \
 	__raw_i915_write##x(dev_priv, reg, val); \
-	REG_WRITE_FOOTER; \
+	GEN2_WRITE_FOOTER; \
 }
 
 #define __gen5_write(x) \
 static void \
 gen5_write##x(struct drm_i915_private *dev_priv, off_t reg, u##x val, bool trace) { \
-	REG_WRITE_HEADER; \
+	GEN2_WRITE_HEADER; \
 	ilk_dummy_write(dev_priv); \
 	__raw_i915_write##x(dev_priv, reg, val); \
-	REG_WRITE_FOOTER; \
+	GEN2_WRITE_FOOTER; \
 }
 
+__gen5_write(8)
+__gen5_write(16)
+__gen5_write(32)
+__gen5_write(64)
+__gen2_write(8)
+__gen2_write(16)
+__gen2_write(32)
+__gen2_write(64)
+
+#undef __gen5_write
+#undef __gen2_write
+
+#undef GEN2_WRITE_FOOTER
+#undef GEN2_WRITE_HEADER
+
+#define GEN6_WRITE_HEADER \
+	unsigned long irqflags; \
+	trace_i915_reg_rw(true, reg, val, sizeof(val), trace); \
+	assert_device_not_suspended(dev_priv); \
+	spin_lock_irqsave(&dev_priv->uncore.lock, irqflags)
+
+#define GEN6_WRITE_FOOTER \
+	spin_unlock_irqrestore(&dev_priv->uncore.lock, irqflags)
+
 #define __gen6_write(x) \
 static void \
 gen6_write##x(struct drm_i915_private *dev_priv, off_t reg, u##x val, bool trace) { \
 	u32 __fifo_ret = 0; \
-	REG_WRITE_HEADER; \
+	GEN6_WRITE_HEADER; \
 	if (NEEDS_FORCE_WAKE((dev_priv), (reg))) { \
 		__fifo_ret = __gen6_gt_wait_for_fifo(dev_priv); \
 	} \
@@ -900,14 +934,14 @@ gen6_write##x(struct drm_i915_private *dev_priv, off_t reg, u##x val, bool trace
 	if (unlikely(__fifo_ret)) { \
 		gen6_gt_check_fifodbg(dev_priv); \
 	} \
-	REG_WRITE_FOOTER; \
+	GEN6_WRITE_FOOTER; \
 }
 
 #define __hsw_write(x) \
 static void \
 hsw_write##x(struct drm_i915_private *dev_priv, off_t reg, u##x val, bool trace) { \
 	u32 __fifo_ret = 0; \
-	REG_WRITE_HEADER; \
+	GEN6_WRITE_HEADER; \
 	if (NEEDS_FORCE_WAKE((dev_priv), (reg))) { \
 		__fifo_ret = __gen6_gt_wait_for_fifo(dev_priv); \
 	} \
@@ -918,7 +952,7 @@ hsw_write##x(struct drm_i915_private *dev_priv, off_t reg, u##x val, bool trace)
 	} \
 	hsw_unclaimed_reg_debug(dev_priv, reg, false, false); \
 	hsw_unclaimed_reg_detect(dev_priv); \
-	REG_WRITE_FOOTER; \
+	GEN6_WRITE_FOOTER; \
 }
 
 static const u32 gen8_shadowed_regs[] = {
@@ -945,7 +979,7 @@ static bool is_gen8_shadowed(struct drm_i915_private *dev_priv, u32 reg)
 #define __gen8_write(x) \
 static void \
 gen8_write##x(struct drm_i915_private *dev_priv, off_t reg, u##x val, bool trace) { \
-	REG_WRITE_HEADER; \
+	GEN6_WRITE_HEADER; \
 	hsw_unclaimed_reg_debug(dev_priv, reg, false, true); \
 	if (reg < 0x40000 && !is_gen8_shadowed(dev_priv, reg)) { \
 		if (dev_priv->uncore.forcewake_count == 0) \
@@ -960,7 +994,7 @@ gen8_write##x(struct drm_i915_private *dev_priv, off_t reg, u##x val, bool trace
 	} \
 	hsw_unclaimed_reg_debug(dev_priv, reg, false, false); \
 	hsw_unclaimed_reg_detect(dev_priv); \
-	REG_WRITE_FOOTER; \
+	GEN6_WRITE_FOOTER; \
 }
 
 #define __chv_write(x) \
@@ -968,7 +1002,7 @@ static void \
 chv_write##x(struct drm_i915_private *dev_priv, off_t reg, u##x val, bool trace) { \
 	unsigned fwengine = 0; \
 	bool shadowed = is_gen8_shadowed(dev_priv, reg); \
-	REG_WRITE_HEADER; \
+	GEN6_WRITE_HEADER; \
 	if (!shadowed) { \
 		if (FORCEWAKE_CHV_RENDER_RANGE_OFFSET(reg)) { \
 			if (dev_priv->uncore.fw_rendercount == 0) \
@@ -988,7 +1022,7 @@ chv_write##x(struct drm_i915_private *dev_priv, off_t reg, u##x val, bool trace)
 	__raw_i915_write##x(dev_priv, reg, val); \
 	if (fwengine) \
 		dev_priv->uncore.funcs.force_wake_put(dev_priv, fwengine); \
-	REG_WRITE_FOOTER; \
+	GEN6_WRITE_FOOTER; \
 }
 
 static const u32 gen9_shadowed_regs[] = {
@@ -1018,7 +1052,7 @@ static bool is_gen9_shadowed(struct drm_i915_private *dev_priv, u32 reg)
 static void \
 gen9_write##x(struct drm_i915_private *dev_priv, off_t reg, u##x val, \
 		bool trace) { \
-	REG_WRITE_HEADER; \
+	GEN6_WRITE_HEADER; \
 	if (!SKL_NEEDS_FORCE_WAKE((dev_priv), (reg)) || \
 			is_gen9_shadowed(dev_priv, reg)) { \
 		__raw_i915_write##x(dev_priv, reg, val); \
@@ -1047,7 +1081,7 @@ gen9_write##x(struct drm_i915_private *dev_priv, off_t reg, u##x val, \
 			dev_priv->uncore.funcs.force_wake_put(dev_priv, \
 					fwengine); \
 	} \
-	REG_WRITE_FOOTER; \
+	GEN6_WRITE_FOOTER; \
 }
 
 __gen9_write(8)
@@ -1070,24 +1104,14 @@ __gen6_write(8)
 __gen6_write(16)
 __gen6_write(32)
 __gen6_write(64)
-__gen5_write(8)
-__gen5_write(16)
-__gen5_write(32)
-__gen5_write(64)
-__gen4_write(8)
-__gen4_write(16)
-__gen4_write(32)
-__gen4_write(64)
 
 #undef __gen9_write
 #undef __chv_write
 #undef __gen8_write
 #undef __hsw_write
 #undef __gen6_write
-#undef __gen5_write
-#undef __gen4_write
-#undef REG_WRITE_FOOTER
-#undef REG_WRITE_HEADER
+#undef GEN6_WRITE_FOOTER
+#undef GEN6_WRITE_HEADER
 
 #define ASSIGN_WRITE_MMIO_VFUNCS(x) \
 do { \
@@ -1200,8 +1224,8 @@ void intel_uncore_init(struct drm_device *dev)
 	case 4:
 	case 3:
 	case 2:
-		ASSIGN_WRITE_MMIO_VFUNCS(gen4);
-		ASSIGN_READ_MMIO_VFUNCS(gen4);
+		ASSIGN_WRITE_MMIO_VFUNCS(gen2);
+		ASSIGN_READ_MMIO_VFUNCS(gen2);
 		break;
 	}
 
-- 
cgit v1.2.3


From b2cff0dbbb7ef03e08865b906b236a06c6cf2e2f Mon Sep 17 00:00:00 2001
From: Chris Wilson <chris@chris-wilson.co.uk>
Date: Fri, 16 Jan 2015 11:34:37 +0200
Subject: drm/i915: Reduce duplicated forcewake logic

Introduce a structure to track the individual forcewake domains and use
that to eliminate duplicate logic.

v2: - Rebase on latest dinq (Mika)
    - for_each_fw_domain macro (Mika)
    - Handle reset atomically, keeping the timer running (Mika)
    - for_each_fw_domain parameter ordering (Chris)
    - defer timer on new register access (Mika)

v3: - Fix forcewake_reset/get race by waiting pending timers

v4: - cond_resched and verbose warning on timer deletion (Chris)
    - need to run pending timers manually on reset

Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> (v1)
Signed-off-by: Mika Kuoppala <mika.kuoppala@intel.com>
Acked-by: Deepak S <deepak.s@linux.intel.com> (v2)
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
---
 drivers/gpu/drm/i915/i915_debugfs.c |  65 +++---
 drivers/gpu/drm/i915/i915_drv.h     |  54 +++--
 drivers/gpu/drm/i915/intel_uncore.c | 445 +++++++++++++++---------------------
 3 files changed, 246 insertions(+), 318 deletions(-)

(limited to 'drivers/gpu/drm/i915/intel_uncore.c')

diff --git a/drivers/gpu/drm/i915/i915_debugfs.c b/drivers/gpu/drm/i915/i915_debugfs.c
index ac517eb07c00..4d156e65837f 100644
--- a/drivers/gpu/drm/i915/i915_debugfs.c
+++ b/drivers/gpu/drm/i915/i915_debugfs.c
@@ -1288,14 +1288,36 @@ static int ironlake_drpc_info(struct seq_file *m)
 	return 0;
 }
 
-static int vlv_drpc_info(struct seq_file *m)
+static int i915_gen6_forcewake_count_info(struct seq_file *m, void *data)
 {
+	struct drm_info_node *node = m->private;
+	struct drm_device *dev = node->minor->dev;
+	struct drm_i915_private *dev_priv = dev->dev_private;
+	struct intel_uncore_forcewake_domain *fw_domain;
+	const char *domain_names[] = {
+		"render",
+		"blitter",
+		"media",
+	};
+	int i;
+
+	spin_lock_irq(&dev_priv->uncore.lock);
+	for_each_fw_domain(fw_domain, dev_priv, i) {
+		seq_printf(m, "%s.wake_count = %u\n",
+			   domain_names[i],
+			   fw_domain->wake_count);
+	}
+	spin_unlock_irq(&dev_priv->uncore.lock);
 
+	return 0;
+}
+
+static int vlv_drpc_info(struct seq_file *m)
+{
 	struct drm_info_node *node = m->private;
 	struct drm_device *dev = node->minor->dev;
 	struct drm_i915_private *dev_priv = dev->dev_private;
 	u32 rpmodectl1, rcctl1, pw_status;
-	unsigned fw_rendercount = 0, fw_mediacount = 0;
 
 	intel_runtime_pm_get(dev_priv);
 
@@ -1327,22 +1349,11 @@ static int vlv_drpc_info(struct seq_file *m)
 	seq_printf(m, "Media RC6 residency since boot: %u\n",
 		   I915_READ(VLV_GT_MEDIA_RC6));
 
-	spin_lock_irq(&dev_priv->uncore.lock);
-	fw_rendercount = dev_priv->uncore.fw_rendercount;
-	fw_mediacount = dev_priv->uncore.fw_mediacount;
-	spin_unlock_irq(&dev_priv->uncore.lock);
-
-	seq_printf(m, "Forcewake Render Count = %u\n", fw_rendercount);
-	seq_printf(m, "Forcewake Media Count = %u\n", fw_mediacount);
-
-
-	return 0;
+	return i915_gen6_forcewake_count_info(m, NULL);
 }
 
-
 static int gen6_drpc_info(struct seq_file *m)
 {
-
 	struct drm_info_node *node = m->private;
 	struct drm_device *dev = node->minor->dev;
 	struct drm_i915_private *dev_priv = dev->dev_private;
@@ -1356,7 +1367,7 @@ static int gen6_drpc_info(struct seq_file *m)
 	intel_runtime_pm_get(dev_priv);
 
 	spin_lock_irq(&dev_priv->uncore.lock);
-	forcewake_count = dev_priv->uncore.forcewake_count;
+	forcewake_count = dev_priv->uncore.fw_domain[FW_DOMAIN_ID_RENDER].wake_count;
 	spin_unlock_irq(&dev_priv->uncore.lock);
 
 	if (forcewake_count) {
@@ -1984,30 +1995,6 @@ static int i915_execlists(struct seq_file *m, void *data)
 	return 0;
 }
 
-static int i915_gen6_forcewake_count_info(struct seq_file *m, void *data)
-{
-	struct drm_info_node *node = m->private;
-	struct drm_device *dev = node->minor->dev;
-	struct drm_i915_private *dev_priv = dev->dev_private;
-	unsigned forcewake_count = 0, fw_rendercount = 0, fw_mediacount = 0;
-
-	spin_lock_irq(&dev_priv->uncore.lock);
-	if (IS_VALLEYVIEW(dev)) {
-		fw_rendercount = dev_priv->uncore.fw_rendercount;
-		fw_mediacount = dev_priv->uncore.fw_mediacount;
-	} else
-		forcewake_count = dev_priv->uncore.forcewake_count;
-	spin_unlock_irq(&dev_priv->uncore.lock);
-
-	if (IS_VALLEYVIEW(dev)) {
-		seq_printf(m, "fw_rendercount = %u\n", fw_rendercount);
-		seq_printf(m, "fw_mediacount = %u\n", fw_mediacount);
-	} else
-		seq_printf(m, "forcewake count = %u\n", forcewake_count);
-
-	return 0;
-}
-
 static const char *swizzle_string(unsigned swizzle)
 {
 	switch (swizzle) {
diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index 3a32e0d81a9f..ac2082b022f3 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -596,20 +596,45 @@ struct intel_uncore_funcs {
 				uint64_t val, bool trace);
 };
 
+enum {
+	FW_DOMAIN_ID_RENDER = 0,
+	FW_DOMAIN_ID_BLITTER,
+	FW_DOMAIN_ID_MEDIA,
+
+	FW_DOMAIN_ID_COUNT
+};
+
 struct intel_uncore {
 	spinlock_t lock; /** lock is also taken in irq contexts. */
 
 	struct intel_uncore_funcs funcs;
 
 	unsigned fifo_count;
-	unsigned forcewake_count;
-
-	unsigned fw_rendercount;
-	unsigned fw_mediacount;
-	unsigned fw_blittercount;
-
-	struct timer_list force_wake_timer;
-};
+	unsigned fw_domains;
+
+	struct intel_uncore_forcewake_domain {
+		struct drm_i915_private *i915;
+		int id;
+		unsigned wake_count;
+		struct timer_list timer;
+	} fw_domain[FW_DOMAIN_ID_COUNT];
+#define FORCEWAKE_RENDER	(1 << FW_DOMAIN_ID_RENDER)
+#define FORCEWAKE_BLITTER	(1 << FW_DOMAIN_ID_BLITTER)
+#define FORCEWAKE_MEDIA		(1 << FW_DOMAIN_ID_MEDIA)
+#define FORCEWAKE_ALL		(FORCEWAKE_RENDER | \
+				 FORCEWAKE_BLITTER | \
+				 FORCEWAKE_MEDIA)
+};
+
+/* Iterate over initialised fw domains */
+#define for_each_fw_domain_mask(domain__, mask__, dev_priv__, i__) \
+	for ((i__) = 0, (domain__) = &(dev_priv__)->uncore.fw_domain[0]; \
+	     (i__) < FW_DOMAIN_ID_COUNT; \
+	     (i__)++, (domain__) = &(dev_priv__)->uncore.fw_domain[i__]) \
+		if (((mask__) & (dev_priv__)->uncore.fw_domains) & (1 << (i__)))
+
+#define for_each_fw_domain(domain__, dev_priv__, i__) \
+	for_each_fw_domain_mask(domain__, FORCEWAKE_ALL, dev_priv__, i__)
 
 #define DEV_INFO_FOR_EACH_FLAG(func, sep) \
 	func(is_mobile) sep \
@@ -3167,8 +3192,10 @@ extern void intel_display_print_error_state(struct drm_i915_error_state_buf *e,
  * must be set to prevent GT core from power down and stale values being
  * returned.
  */
-void gen6_gt_force_wake_get(struct drm_i915_private *dev_priv, int fw_engine);
-void gen6_gt_force_wake_put(struct drm_i915_private *dev_priv, int fw_engine);
+void gen6_gt_force_wake_get(struct drm_i915_private *dev_priv,
+			    unsigned fw_domains);
+void gen6_gt_force_wake_put(struct drm_i915_private *dev_priv,
+			    unsigned fw_domains);
 void assert_force_wake_inactive(struct drm_i915_private *dev_priv);
 
 int sandybridge_pcode_read(struct drm_i915_private *dev_priv, u32 mbox, u32 *val);
@@ -3200,13 +3227,6 @@ void vlv_flisdsi_write(struct drm_i915_private *dev_priv, u32 reg, u32 val);
 int vlv_gpu_freq(struct drm_i915_private *dev_priv, int val);
 int vlv_freq_opcode(struct drm_i915_private *dev_priv, int val);
 
-#define FORCEWAKE_RENDER	(1 << 0)
-#define FORCEWAKE_MEDIA		(1 << 1)
-#define FORCEWAKE_BLITTER	(1 << 2)
-#define FORCEWAKE_ALL		(FORCEWAKE_RENDER | FORCEWAKE_MEDIA | \
-					FORCEWAKE_BLITTER)
-
-
 #define I915_READ8(reg)		dev_priv->uncore.funcs.mmio_readb(dev_priv, (reg), true)
 #define I915_WRITE8(reg, val)	dev_priv->uncore.funcs.mmio_writeb(dev_priv, (reg), (val), true)
 
diff --git a/drivers/gpu/drm/i915/intel_uncore.c b/drivers/gpu/drm/i915/intel_uncore.c
index 3d1ffacb7612..b35f3a9e6869 100644
--- a/drivers/gpu/drm/i915/intel_uncore.c
+++ b/drivers/gpu/drm/i915/intel_uncore.c
@@ -67,7 +67,7 @@ static void __gen6_gt_force_wake_reset(struct drm_i915_private *dev_priv)
 }
 
 static void __gen6_gt_force_wake_get(struct drm_i915_private *dev_priv,
-							int fw_engine)
+				     int fw_engine)
 {
 	if (wait_for_atomic((__raw_i915_read32(dev_priv, FORCEWAKE_ACK) & 1) == 0,
 			    FORCEWAKE_ACK_TIMEOUT_MS))
@@ -93,7 +93,7 @@ static void __gen7_gt_force_wake_mt_reset(struct drm_i915_private *dev_priv)
 }
 
 static void __gen7_gt_force_wake_mt_get(struct drm_i915_private *dev_priv,
-							int fw_engine)
+					int fw_engine)
 {
 	u32 forcewake_ack;
 
@@ -129,7 +129,7 @@ static void gen6_gt_check_fifodbg(struct drm_i915_private *dev_priv)
 }
 
 static void __gen6_gt_force_wake_put(struct drm_i915_private *dev_priv,
-							int fw_engine)
+				     int fw_engine)
 {
 	__raw_i915_write32(dev_priv, FORCEWAKE, 0);
 	/* something from same cacheline, but !FORCEWAKE */
@@ -138,7 +138,7 @@ static void __gen6_gt_force_wake_put(struct drm_i915_private *dev_priv,
 }
 
 static void __gen7_gt_force_wake_mt_put(struct drm_i915_private *dev_priv,
-							int fw_engine)
+					int fw_engine)
 {
 	__raw_i915_write32(dev_priv, FORCEWAKE_MT,
 			   _MASKED_BIT_DISABLE(FORCEWAKE_KERNEL));
@@ -187,7 +187,7 @@ static void vlv_force_wake_reset(struct drm_i915_private *dev_priv)
 }
 
 static void __vlv_force_wake_get(struct drm_i915_private *dev_priv,
-						int fw_engine)
+				 int fw_engine)
 {
 	/* Check for Render Engine */
 	if (FORCEWAKE_RENDER & fw_engine) {
@@ -227,9 +227,8 @@ static void __vlv_force_wake_get(struct drm_i915_private *dev_priv,
 }
 
 static void __vlv_force_wake_put(struct drm_i915_private *dev_priv,
-					int fw_engine)
+				 int fw_engine)
 {
-
 	/* Check for Render Engine */
 	if (FORCEWAKE_RENDER & fw_engine)
 		__raw_i915_write32(dev_priv, FORCEWAKE_VLV,
@@ -247,37 +246,6 @@ static void __vlv_force_wake_put(struct drm_i915_private *dev_priv,
 		gen6_gt_check_fifodbg(dev_priv);
 }
 
-static void vlv_force_wake_get(struct drm_i915_private *dev_priv, int fw_engine)
-{
-	if (fw_engine & FORCEWAKE_RENDER &&
-	    dev_priv->uncore.fw_rendercount++ != 0)
-		fw_engine &= ~FORCEWAKE_RENDER;
-	if (fw_engine & FORCEWAKE_MEDIA &&
-	    dev_priv->uncore.fw_mediacount++ != 0)
-		fw_engine &= ~FORCEWAKE_MEDIA;
-
-	if (fw_engine)
-		dev_priv->uncore.funcs.force_wake_get(dev_priv, fw_engine);
-}
-
-static void vlv_force_wake_put(struct drm_i915_private *dev_priv, int fw_engine)
-{
-	if (fw_engine & FORCEWAKE_RENDER) {
-		WARN_ON(!dev_priv->uncore.fw_rendercount);
-		if (--dev_priv->uncore.fw_rendercount != 0)
-			fw_engine &= ~FORCEWAKE_RENDER;
-	}
-
-	if (fw_engine & FORCEWAKE_MEDIA) {
-		WARN_ON(!dev_priv->uncore.fw_mediacount);
-		if (--dev_priv->uncore.fw_mediacount != 0)
-			fw_engine &= ~FORCEWAKE_MEDIA;
-	}
-
-	if (fw_engine)
-		dev_priv->uncore.funcs.force_wake_put(dev_priv, fw_engine);
-}
-
 static void __gen9_gt_force_wake_mt_reset(struct drm_i915_private *dev_priv)
 {
 	__raw_i915_write32(dev_priv, FORCEWAKE_RENDER_GEN9,
@@ -367,80 +335,72 @@ __gen9_force_wake_put(struct drm_i915_private *dev_priv, int fw_engine)
 				_MASKED_BIT_DISABLE(FORCEWAKE_KERNEL));
 }
 
-static void
-gen9_force_wake_get(struct drm_i915_private *dev_priv, int fw_engine)
+static void gen6_force_wake_timer(unsigned long arg)
 {
-	if (FORCEWAKE_RENDER & fw_engine) {
-		if (dev_priv->uncore.fw_rendercount++ == 0)
-			dev_priv->uncore.funcs.force_wake_get(dev_priv,
-							FORCEWAKE_RENDER);
-	}
+	struct intel_uncore_forcewake_domain *domain = (void *)arg;
+	unsigned long irqflags;
 
-	if (FORCEWAKE_MEDIA & fw_engine) {
-		if (dev_priv->uncore.fw_mediacount++ == 0)
-			dev_priv->uncore.funcs.force_wake_get(dev_priv,
-							FORCEWAKE_MEDIA);
-	}
+	assert_device_not_suspended(domain->i915);
 
-	if (FORCEWAKE_BLITTER & fw_engine) {
-		if (dev_priv->uncore.fw_blittercount++ == 0)
-			dev_priv->uncore.funcs.force_wake_get(dev_priv,
-							FORCEWAKE_BLITTER);
-	}
+	spin_lock_irqsave(&domain->i915->uncore.lock, irqflags);
+	if (WARN_ON(domain->wake_count == 0))
+		domain->wake_count++;
+
+	if (--domain->wake_count == 0)
+		domain->i915->uncore.funcs.force_wake_put(domain->i915,
+							  1 << domain->id);
+
+	spin_unlock_irqrestore(&domain->i915->uncore.lock, irqflags);
 }
 
-static void
-gen9_force_wake_put(struct drm_i915_private *dev_priv, int fw_engine)
+void intel_uncore_forcewake_reset(struct drm_device *dev, bool restore)
 {
-	if (FORCEWAKE_RENDER & fw_engine) {
-		WARN_ON(dev_priv->uncore.fw_rendercount == 0);
-		if (--dev_priv->uncore.fw_rendercount == 0)
-			dev_priv->uncore.funcs.force_wake_put(dev_priv,
-							FORCEWAKE_RENDER);
-	}
+	struct drm_i915_private *dev_priv = dev->dev_private;
+	unsigned long irqflags, fw = 0;
+	struct intel_uncore_forcewake_domain *domain;
+	int id, active_domains, retry_count = 100;
 
-	if (FORCEWAKE_MEDIA & fw_engine) {
-		WARN_ON(dev_priv->uncore.fw_mediacount == 0);
-		if (--dev_priv->uncore.fw_mediacount == 0)
-			dev_priv->uncore.funcs.force_wake_put(dev_priv,
-							FORCEWAKE_MEDIA);
-	}
+	/* Hold uncore.lock across reset to prevent any register access
+	 * with forcewake not set correctly. Wait until all pending
+	 * timers are run before holding.
+	 */
+	while (1) {
+		active_domains = 0;
 
-	if (FORCEWAKE_BLITTER & fw_engine) {
-		WARN_ON(dev_priv->uncore.fw_blittercount == 0);
-		if (--dev_priv->uncore.fw_blittercount == 0)
-			dev_priv->uncore.funcs.force_wake_put(dev_priv,
-							FORCEWAKE_BLITTER);
-	}
-}
+		for_each_fw_domain(domain, dev_priv, id) {
+			if (del_timer_sync(&domain->timer) == 0)
+				continue;
 
-static void gen6_force_wake_timer(unsigned long arg)
-{
-	struct drm_i915_private *dev_priv = (void *)arg;
-	unsigned long irqflags;
+			gen6_force_wake_timer((unsigned long)domain);
+		}
 
-	assert_device_not_suspended(dev_priv);
+		spin_lock_irqsave(&dev_priv->uncore.lock, irqflags);
 
-	spin_lock_irqsave(&dev_priv->uncore.lock, irqflags);
-	WARN_ON(!dev_priv->uncore.forcewake_count);
+		for_each_fw_domain(domain, dev_priv, id) {
+			if (timer_pending(&domain->timer))
+				active_domains |= (1 << id);
+		}
 
-	if (--dev_priv->uncore.forcewake_count == 0)
-		dev_priv->uncore.funcs.force_wake_put(dev_priv, FORCEWAKE_ALL);
-	spin_unlock_irqrestore(&dev_priv->uncore.lock, irqflags);
-}
+		if (active_domains == 0)
+			break;
 
-void intel_uncore_forcewake_reset(struct drm_device *dev, bool restore)
-{
-	struct drm_i915_private *dev_priv = dev->dev_private;
-	unsigned long irqflags;
+		if (--retry_count == 0) {
+			DRM_ERROR("Timed out waiting for forcewake timers to finish\n");
+			break;
+		}
 
-	if (del_timer_sync(&dev_priv->uncore.force_wake_timer))
-		gen6_force_wake_timer((unsigned long)dev_priv);
+		spin_unlock_irqrestore(&dev_priv->uncore.lock, irqflags);
+		cond_resched();
+	}
 
-	/* Hold uncore.lock across reset to prevent any register access
-	 * with forcewake not set correctly
-	 */
-	spin_lock_irqsave(&dev_priv->uncore.lock, irqflags);
+	WARN_ON(active_domains);
+
+	for_each_fw_domain(domain, dev_priv, id)
+		if (domain->wake_count)
+			fw |= 1 << id;
+
+	if (fw)
+		dev_priv->uncore.funcs.force_wake_put(dev_priv, fw);
 
 	if (IS_VALLEYVIEW(dev))
 		vlv_force_wake_reset(dev_priv);
@@ -454,28 +414,6 @@ void intel_uncore_forcewake_reset(struct drm_device *dev, bool restore)
 		__gen9_gt_force_wake_mt_reset(dev_priv);
 
 	if (restore) { /* If reset with a user forcewake, try to restore */
-		unsigned fw = 0;
-
-		if (IS_VALLEYVIEW(dev)) {
-			if (dev_priv->uncore.fw_rendercount)
-				fw |= FORCEWAKE_RENDER;
-
-			if (dev_priv->uncore.fw_mediacount)
-				fw |= FORCEWAKE_MEDIA;
-		} else if (IS_GEN9(dev)) {
-			if (dev_priv->uncore.fw_rendercount)
-				fw |= FORCEWAKE_RENDER;
-
-			if (dev_priv->uncore.fw_mediacount)
-				fw |= FORCEWAKE_MEDIA;
-
-			if (dev_priv->uncore.fw_blittercount)
-				fw |= FORCEWAKE_BLITTER;
-		} else {
-			if (dev_priv->uncore.forcewake_count)
-				fw = FORCEWAKE_ALL;
-		}
-
 		if (fw)
 			dev_priv->uncore.funcs.force_wake_get(dev_priv, fw);
 
@@ -485,6 +423,9 @@ void intel_uncore_forcewake_reset(struct drm_device *dev, bool restore)
 				GT_FIFO_FREE_ENTRIES_MASK;
 	}
 
+	if (!restore)
+		assert_force_wake_inactive(dev_priv);
+
 	spin_unlock_irqrestore(&dev_priv->uncore.lock, irqflags);
 }
 
@@ -533,53 +474,59 @@ void intel_uncore_sanitize(struct drm_device *dev)
  * be called at the beginning of the sequence followed by a call to
  * gen6_gt_force_wake_put() at the end of the sequence.
  */
-void gen6_gt_force_wake_get(struct drm_i915_private *dev_priv, int fw_engine)
+void gen6_gt_force_wake_get(struct drm_i915_private *dev_priv,
+			    unsigned fw_domains)
 {
 	unsigned long irqflags;
+	struct intel_uncore_forcewake_domain *domain;
+	int id;
 
 	if (!dev_priv->uncore.funcs.force_wake_get)
 		return;
 
 	WARN_ON(dev_priv->pm.suspended);
 
+	fw_domains &= dev_priv->uncore.fw_domains;
+
 	spin_lock_irqsave(&dev_priv->uncore.lock, irqflags);
 
-	if (IS_GEN9(dev_priv->dev)) {
-		gen9_force_wake_get(dev_priv, fw_engine);
-	} else if (IS_VALLEYVIEW(dev_priv->dev)) {
-		vlv_force_wake_get(dev_priv, fw_engine);
-	} else {
-		if (dev_priv->uncore.forcewake_count++ == 0)
-			dev_priv->uncore.funcs.force_wake_get(dev_priv,
-							      FORCEWAKE_ALL);
+	for_each_fw_domain_mask(domain, fw_domains, dev_priv, id) {
+		if (domain->wake_count++)
+			fw_domains &= ~(1 << id);
 	}
 
+	if (fw_domains)
+		dev_priv->uncore.funcs.force_wake_get(dev_priv, fw_domains);
+
 	spin_unlock_irqrestore(&dev_priv->uncore.lock, irqflags);
 }
 
 /*
  * see gen6_gt_force_wake_get()
  */
-void gen6_gt_force_wake_put(struct drm_i915_private *dev_priv, int fw_engine)
+void gen6_gt_force_wake_put(struct drm_i915_private *dev_priv,
+			    unsigned fw_domains)
 {
 	unsigned long irqflags;
+	struct intel_uncore_forcewake_domain *domain;
+	int id;
 
 	if (!dev_priv->uncore.funcs.force_wake_put)
 		return;
 
+	fw_domains &= dev_priv->uncore.fw_domains;
+
 	spin_lock_irqsave(&dev_priv->uncore.lock, irqflags);
 
-	if (IS_GEN9(dev_priv->dev)) {
-		gen9_force_wake_put(dev_priv, fw_engine);
-	} else if (IS_VALLEYVIEW(dev_priv->dev)) {
-		vlv_force_wake_put(dev_priv, fw_engine);
-	} else {
-		WARN_ON(!dev_priv->uncore.forcewake_count);
-		if (--dev_priv->uncore.forcewake_count == 0) {
-			dev_priv->uncore.forcewake_count++;
-			mod_timer_pinned(&dev_priv->uncore.force_wake_timer,
-					 jiffies + 1);
-		}
+	for_each_fw_domain_mask(domain, fw_domains, dev_priv, id) {
+		if (WARN_ON(domain->wake_count == 0))
+			continue;
+
+		if (--domain->wake_count)
+			continue;
+
+		domain->wake_count++;
+		mod_timer_pinned(&domain->timer, jiffies + 1);
 	}
 
 	spin_unlock_irqrestore(&dev_priv->uncore.lock, irqflags);
@@ -587,10 +534,14 @@ void gen6_gt_force_wake_put(struct drm_i915_private *dev_priv, int fw_engine)
 
 void assert_force_wake_inactive(struct drm_i915_private *dev_priv)
 {
+	struct intel_uncore_forcewake_domain *domain;
+	int i;
+
 	if (!dev_priv->uncore.funcs.force_wake_get)
 		return;
 
-	WARN_ON(dev_priv->uncore.forcewake_count > 0);
+	for_each_fw_domain(domain, dev_priv, i)
+		WARN_ON(domain->wake_count);
 }
 
 /* We give fast paths for the really cool registers */
@@ -753,19 +704,37 @@ __gen2_read(64)
 	trace_i915_reg_rw(false, reg, val, sizeof(val), trace); \
 	return val
 
+static inline void __force_wake_get(struct drm_i915_private *dev_priv,
+				    unsigned fw_domains)
+{
+	struct intel_uncore_forcewake_domain *domain;
+	int i;
+
+	if (WARN_ON(!fw_domains))
+		return;
+
+	/* Ideally GCC would be constant-fold and eliminate this loop */
+	for_each_fw_domain_mask(domain, fw_domains, dev_priv, i) {
+		if (domain->wake_count) {
+			fw_domains &= ~(1 << i);
+			continue;
+		}
+
+		domain->wake_count++;
+		mod_timer_pinned(&domain->timer, jiffies + 1);
+	}
+
+	if (fw_domains)
+		dev_priv->uncore.funcs.force_wake_get(dev_priv, fw_domains);
+}
+
 #define __gen6_read(x) \
 static u##x \
 gen6_read##x(struct drm_i915_private *dev_priv, off_t reg, bool trace) { \
 	GEN6_READ_HEADER(x); \
 	hsw_unclaimed_reg_debug(dev_priv, reg, true, true); \
-	if (dev_priv->uncore.forcewake_count == 0 && \
-	    NEEDS_FORCE_WAKE((dev_priv), (reg))) { \
-		dev_priv->uncore.funcs.force_wake_get(dev_priv, \
-						      FORCEWAKE_ALL); \
-		dev_priv->uncore.forcewake_count++; \
-		mod_timer_pinned(&dev_priv->uncore.force_wake_timer, \
-				 jiffies + 1); \
-	} \
+	if (NEEDS_FORCE_WAKE((dev_priv), (reg))) \
+		__force_wake_get(dev_priv, FORCEWAKE_RENDER); \
 	val = __raw_i915_read##x(dev_priv, reg); \
 	hsw_unclaimed_reg_debug(dev_priv, reg, true, false); \
 	GEN6_READ_FOOTER; \
@@ -774,45 +743,27 @@ gen6_read##x(struct drm_i915_private *dev_priv, off_t reg, bool trace) { \
 #define __vlv_read(x) \
 static u##x \
 vlv_read##x(struct drm_i915_private *dev_priv, off_t reg, bool trace) { \
-	unsigned fwengine = 0; \
 	GEN6_READ_HEADER(x); \
-	if (FORCEWAKE_VLV_RENDER_RANGE_OFFSET(reg)) { \
-		if (dev_priv->uncore.fw_rendercount == 0) \
-			fwengine = FORCEWAKE_RENDER; \
-	} else if (FORCEWAKE_VLV_MEDIA_RANGE_OFFSET(reg)) { \
-		if (dev_priv->uncore.fw_mediacount == 0) \
-			fwengine = FORCEWAKE_MEDIA; \
-	}  \
-	if (fwengine) \
-		dev_priv->uncore.funcs.force_wake_get(dev_priv, fwengine); \
+	if (FORCEWAKE_VLV_RENDER_RANGE_OFFSET(reg)) \
+		__force_wake_get(dev_priv, FORCEWAKE_RENDER); \
+	else if (FORCEWAKE_VLV_MEDIA_RANGE_OFFSET(reg)) \
+		__force_wake_get(dev_priv, FORCEWAKE_MEDIA); \
 	val = __raw_i915_read##x(dev_priv, reg); \
-	if (fwengine) \
-		dev_priv->uncore.funcs.force_wake_put(dev_priv, fwengine); \
 	GEN6_READ_FOOTER; \
 }
 
 #define __chv_read(x) \
 static u##x \
 chv_read##x(struct drm_i915_private *dev_priv, off_t reg, bool trace) { \
-	unsigned fwengine = 0; \
 	GEN6_READ_HEADER(x); \
-	if (FORCEWAKE_CHV_RENDER_RANGE_OFFSET(reg)) { \
-		if (dev_priv->uncore.fw_rendercount == 0) \
-			fwengine = FORCEWAKE_RENDER; \
-	} else if (FORCEWAKE_CHV_MEDIA_RANGE_OFFSET(reg)) { \
-		if (dev_priv->uncore.fw_mediacount == 0) \
-			fwengine = FORCEWAKE_MEDIA; \
-	} else if (FORCEWAKE_CHV_COMMON_RANGE_OFFSET(reg)) { \
-		if (dev_priv->uncore.fw_rendercount == 0) \
-			fwengine |= FORCEWAKE_RENDER; \
-		if (dev_priv->uncore.fw_mediacount == 0) \
-			fwengine |= FORCEWAKE_MEDIA; \
-	} \
-	if (fwengine) \
-		dev_priv->uncore.funcs.force_wake_get(dev_priv, fwengine); \
+	if (FORCEWAKE_CHV_RENDER_RANGE_OFFSET(reg)) \
+		__force_wake_get(dev_priv, FORCEWAKE_RENDER); \
+	else if (FORCEWAKE_CHV_MEDIA_RANGE_OFFSET(reg)) \
+		__force_wake_get(dev_priv, FORCEWAKE_MEDIA); \
+	else if (FORCEWAKE_CHV_COMMON_RANGE_OFFSET(reg)) \
+		__force_wake_get(dev_priv, \
+				 FORCEWAKE_RENDER | FORCEWAKE_MEDIA); \
 	val = __raw_i915_read##x(dev_priv, reg); \
-	if (fwengine) \
-		dev_priv->uncore.funcs.force_wake_put(dev_priv, fwengine); \
 	GEN6_READ_FOOTER; \
 }
 
@@ -822,32 +773,21 @@ chv_read##x(struct drm_i915_private *dev_priv, off_t reg, bool trace) { \
 #define __gen9_read(x) \
 static u##x \
 gen9_read##x(struct drm_i915_private *dev_priv, off_t reg, bool trace) { \
+	unsigned fw_engine; \
 	GEN6_READ_HEADER(x); \
-	if (!SKL_NEEDS_FORCE_WAKE((dev_priv), (reg))) { \
-		val = __raw_i915_read##x(dev_priv, reg); \
-	} else { \
-		unsigned fwengine = 0; \
-		if (FORCEWAKE_GEN9_RENDER_RANGE_OFFSET(reg)) { \
-			if (dev_priv->uncore.fw_rendercount == 0) \
-				fwengine = FORCEWAKE_RENDER; \
-		} else if (FORCEWAKE_GEN9_MEDIA_RANGE_OFFSET(reg)) { \
-			if (dev_priv->uncore.fw_mediacount == 0) \
-				fwengine = FORCEWAKE_MEDIA; \
-		} else if (FORCEWAKE_GEN9_COMMON_RANGE_OFFSET(reg)) { \
-			if (dev_priv->uncore.fw_rendercount == 0) \
-				fwengine |= FORCEWAKE_RENDER; \
-			if (dev_priv->uncore.fw_mediacount == 0) \
-				fwengine |= FORCEWAKE_MEDIA; \
-		} else { \
-			if (dev_priv->uncore.fw_blittercount == 0) \
-				fwengine = FORCEWAKE_BLITTER; \
-		} \
-		if (fwengine) \
-			dev_priv->uncore.funcs.force_wake_get(dev_priv, fwengine); \
-		val = __raw_i915_read##x(dev_priv, reg); \
-		if (fwengine) \
-			dev_priv->uncore.funcs.force_wake_put(dev_priv, fwengine); \
-	} \
+	if (!SKL_NEEDS_FORCE_WAKE((dev_priv), (reg)))	\
+		fw_engine = 0; \
+	else if (FORCEWAKE_GEN9_RENDER_RANGE_OFFSET(reg))	\
+		fw_engine = FORCEWAKE_RENDER; \
+	else if (FORCEWAKE_GEN9_MEDIA_RANGE_OFFSET(reg)) \
+		fw_engine = FORCEWAKE_MEDIA; \
+	else if (FORCEWAKE_GEN9_COMMON_RANGE_OFFSET(reg)) \
+		fw_engine = FORCEWAKE_RENDER | FORCEWAKE_MEDIA; \
+	else \
+		fw_engine = FORCEWAKE_BLITTER; \
+	if (fw_engine) \
+		__force_wake_get(dev_priv, fw_engine); \
+	val = __raw_i915_read##x(dev_priv, reg); \
 	GEN6_READ_FOOTER; \
 }
 
@@ -981,17 +921,9 @@ static void \
 gen8_write##x(struct drm_i915_private *dev_priv, off_t reg, u##x val, bool trace) { \
 	GEN6_WRITE_HEADER; \
 	hsw_unclaimed_reg_debug(dev_priv, reg, false, true); \
-	if (reg < 0x40000 && !is_gen8_shadowed(dev_priv, reg)) { \
-		if (dev_priv->uncore.forcewake_count == 0) \
-			dev_priv->uncore.funcs.force_wake_get(dev_priv,	\
-							      FORCEWAKE_ALL); \
-		__raw_i915_write##x(dev_priv, reg, val); \
-		if (dev_priv->uncore.forcewake_count == 0) \
-			dev_priv->uncore.funcs.force_wake_put(dev_priv, \
-							      FORCEWAKE_ALL); \
-	} else { \
-		__raw_i915_write##x(dev_priv, reg, val); \
-	} \
+	if (reg < 0x40000 && !is_gen8_shadowed(dev_priv, reg)) \
+		__force_wake_get(dev_priv, FORCEWAKE_RENDER); \
+	__raw_i915_write##x(dev_priv, reg, val); \
 	hsw_unclaimed_reg_debug(dev_priv, reg, false, false); \
 	hsw_unclaimed_reg_detect(dev_priv); \
 	GEN6_WRITE_FOOTER; \
@@ -1000,28 +932,17 @@ gen8_write##x(struct drm_i915_private *dev_priv, off_t reg, u##x val, bool trace
 #define __chv_write(x) \
 static void \
 chv_write##x(struct drm_i915_private *dev_priv, off_t reg, u##x val, bool trace) { \
-	unsigned fwengine = 0; \
 	bool shadowed = is_gen8_shadowed(dev_priv, reg); \
 	GEN6_WRITE_HEADER; \
 	if (!shadowed) { \
-		if (FORCEWAKE_CHV_RENDER_RANGE_OFFSET(reg)) { \
-			if (dev_priv->uncore.fw_rendercount == 0) \
-				fwengine = FORCEWAKE_RENDER; \
-		} else if (FORCEWAKE_CHV_MEDIA_RANGE_OFFSET(reg)) { \
-			if (dev_priv->uncore.fw_mediacount == 0) \
-				fwengine = FORCEWAKE_MEDIA; \
-		} else if (FORCEWAKE_CHV_COMMON_RANGE_OFFSET(reg)) { \
-			if (dev_priv->uncore.fw_rendercount == 0) \
-				fwengine |= FORCEWAKE_RENDER; \
-			if (dev_priv->uncore.fw_mediacount == 0) \
-				fwengine |= FORCEWAKE_MEDIA; \
-		} \
+		if (FORCEWAKE_CHV_RENDER_RANGE_OFFSET(reg)) \
+			__force_wake_get(dev_priv, FORCEWAKE_RENDER); \
+		else if (FORCEWAKE_CHV_MEDIA_RANGE_OFFSET(reg)) \
+			__force_wake_get(dev_priv, FORCEWAKE_MEDIA); \
+		else if (FORCEWAKE_CHV_COMMON_RANGE_OFFSET(reg)) \
+			__force_wake_get(dev_priv, FORCEWAKE_RENDER | FORCEWAKE_MEDIA); \
 	} \
-	if (fwengine) \
-		dev_priv->uncore.funcs.force_wake_get(dev_priv, fwengine); \
 	__raw_i915_write##x(dev_priv, reg, val); \
-	if (fwengine) \
-		dev_priv->uncore.funcs.force_wake_put(dev_priv, fwengine); \
 	GEN6_WRITE_FOOTER; \
 }
 
@@ -1052,35 +973,22 @@ static bool is_gen9_shadowed(struct drm_i915_private *dev_priv, u32 reg)
 static void \
 gen9_write##x(struct drm_i915_private *dev_priv, off_t reg, u##x val, \
 		bool trace) { \
+	unsigned fw_engine; \
 	GEN6_WRITE_HEADER; \
-	if (!SKL_NEEDS_FORCE_WAKE((dev_priv), (reg)) || \
-			is_gen9_shadowed(dev_priv, reg)) { \
-		__raw_i915_write##x(dev_priv, reg, val); \
-	} else { \
-		unsigned fwengine = 0; \
-		if (FORCEWAKE_GEN9_RENDER_RANGE_OFFSET(reg)) { \
-			if (dev_priv->uncore.fw_rendercount == 0) \
-				fwengine = FORCEWAKE_RENDER; \
-		} else if (FORCEWAKE_GEN9_MEDIA_RANGE_OFFSET(reg)) { \
-			if (dev_priv->uncore.fw_mediacount == 0) \
-				fwengine = FORCEWAKE_MEDIA; \
-		} else if (FORCEWAKE_GEN9_COMMON_RANGE_OFFSET(reg)) { \
-			if (dev_priv->uncore.fw_rendercount == 0) \
-				fwengine |= FORCEWAKE_RENDER; \
-			if (dev_priv->uncore.fw_mediacount == 0) \
-				fwengine |= FORCEWAKE_MEDIA; \
-		} else { \
-			if (dev_priv->uncore.fw_blittercount == 0) \
-				fwengine = FORCEWAKE_BLITTER; \
-		} \
-		if (fwengine) \
-			dev_priv->uncore.funcs.force_wake_get(dev_priv, \
-					fwengine); \
-		__raw_i915_write##x(dev_priv, reg, val); \
-		if (fwengine) \
-			dev_priv->uncore.funcs.force_wake_put(dev_priv, \
-					fwengine); \
-	} \
+	if (!SKL_NEEDS_FORCE_WAKE((dev_priv), (reg)) ||	\
+	    is_gen9_shadowed(dev_priv, reg)) \
+		fw_engine = 0; \
+	else if (FORCEWAKE_GEN9_RENDER_RANGE_OFFSET(reg)) \
+		fw_engine = FORCEWAKE_RENDER; \
+	else if (FORCEWAKE_GEN9_MEDIA_RANGE_OFFSET(reg)) \
+		fw_engine = FORCEWAKE_MEDIA; \
+	else if (FORCEWAKE_GEN9_COMMON_RANGE_OFFSET(reg)) \
+		fw_engine = FORCEWAKE_RENDER | FORCEWAKE_MEDIA; \
+	else \
+		fw_engine = FORCEWAKE_BLITTER; \
+	if (fw_engine) \
+		__force_wake_get(dev_priv, fw_engine); \
+	__raw_i915_write##x(dev_priv, reg, val); \
 	GEN6_WRITE_FOOTER; \
 }
 
@@ -1132,21 +1040,24 @@ do { \
 void intel_uncore_init(struct drm_device *dev)
 {
 	struct drm_i915_private *dev_priv = dev->dev_private;
-
-	setup_timer(&dev_priv->uncore.force_wake_timer,
-		    gen6_force_wake_timer, (unsigned long)dev_priv);
+	struct intel_uncore_forcewake_domain *domain;
+	int i;
 
 	__intel_uncore_early_sanitize(dev, false);
 
 	if (IS_GEN9(dev)) {
 		dev_priv->uncore.funcs.force_wake_get = __gen9_force_wake_get;
 		dev_priv->uncore.funcs.force_wake_put = __gen9_force_wake_put;
+		dev_priv->uncore.fw_domains = FORCEWAKE_RENDER |
+			FORCEWAKE_BLITTER | FORCEWAKE_MEDIA;
 	} else if (IS_VALLEYVIEW(dev)) {
 		dev_priv->uncore.funcs.force_wake_get = __vlv_force_wake_get;
 		dev_priv->uncore.funcs.force_wake_put = __vlv_force_wake_put;
+		dev_priv->uncore.fw_domains = FORCEWAKE_RENDER | FORCEWAKE_MEDIA;
 	} else if (IS_HASWELL(dev) || IS_BROADWELL(dev)) {
 		dev_priv->uncore.funcs.force_wake_get = __gen7_gt_force_wake_mt_get;
 		dev_priv->uncore.funcs.force_wake_put = __gen7_gt_force_wake_mt_put;
+		dev_priv->uncore.fw_domains = FORCEWAKE_RENDER;
 	} else if (IS_IVYBRIDGE(dev)) {
 		u32 ecobus;
 
@@ -1178,11 +1089,21 @@ void intel_uncore_init(struct drm_device *dev)
 			dev_priv->uncore.funcs.force_wake_put =
 				__gen6_gt_force_wake_put;
 		}
+		dev_priv->uncore.fw_domains = FORCEWAKE_RENDER;
 	} else if (IS_GEN6(dev)) {
 		dev_priv->uncore.funcs.force_wake_get =
 			__gen6_gt_force_wake_get;
 		dev_priv->uncore.funcs.force_wake_put =
 			__gen6_gt_force_wake_put;
+		dev_priv->uncore.fw_domains = FORCEWAKE_RENDER;
+	}
+
+	for_each_fw_domain(domain, dev_priv, i) {
+		domain->i915 = dev_priv;
+		domain->id = i;
+
+		setup_timer(&domain->timer, gen6_force_wake_timer,
+			    (unsigned long)domain);
 	}
 
 	switch (INTEL_INFO(dev)->gen) {
-- 
cgit v1.2.3


From 05a2fb157e44a53c79133805d30eaada43911941 Mon Sep 17 00:00:00 2001
From: Mika Kuoppala <mika.kuoppala@linux.intel.com>
Date: Mon, 19 Jan 2015 16:20:43 +0200
Subject: drm/i915: Consolidate forcewake code

As we now have forcewake domains, take advantage of it
by putting the differences in gen fw handling in data rather
than in code.

In past we have opencoded this quite extensively as the fw handling
is in the fast path. There has also been a lot of cargo-culted
copy'n'pasting from older gens to newer ones.

Now when the releasing of the forcewake is done by deferred timer,
it gives chance to consolidate more. Due to the frequency of actual hw
access being significantly less.

Take advantage of this and generalize the fw handling code
as much as possible. But we still aim to keep the forcewake sequence
particularities for each gen intact. So the access pattern
to fw engines should remain the same.

v2: - s/old_ack/clear_ack (Chris)
    - s/post_read/posting_read (Chris)
    - less polite commit msg (Chris)

v3: - rebase
    - check and clear wake_count in init

v4: - fix posting reads for gen8 (PRTS)

Signed-off-by: Mika Kuoppala <mika.kuoppala@intel.com>
Reviewed-by: Deepak S <deepak.s@linux.intel.com> (v2)
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
---
 drivers/gpu/drm/i915/i915_debugfs.c |   7 +-
 drivers/gpu/drm/i915/i915_drv.h     |   7 +
 drivers/gpu/drm/i915/intel_uncore.c | 471 ++++++++++++++++--------------------
 3 files changed, 215 insertions(+), 270 deletions(-)

(limited to 'drivers/gpu/drm/i915/intel_uncore.c')

diff --git a/drivers/gpu/drm/i915/i915_debugfs.c b/drivers/gpu/drm/i915/i915_debugfs.c
index 4d156e65837f..db7fcf385cd8 100644
--- a/drivers/gpu/drm/i915/i915_debugfs.c
+++ b/drivers/gpu/drm/i915/i915_debugfs.c
@@ -1294,17 +1294,12 @@ static int i915_gen6_forcewake_count_info(struct seq_file *m, void *data)
 	struct drm_device *dev = node->minor->dev;
 	struct drm_i915_private *dev_priv = dev->dev_private;
 	struct intel_uncore_forcewake_domain *fw_domain;
-	const char *domain_names[] = {
-		"render",
-		"blitter",
-		"media",
-	};
 	int i;
 
 	spin_lock_irq(&dev_priv->uncore.lock);
 	for_each_fw_domain(fw_domain, dev_priv, i) {
 		seq_printf(m, "%s.wake_count = %u\n",
-			   domain_names[i],
+			   intel_uncore_forcewake_domain_to_str(i),
 			   fw_domain->wake_count);
 	}
 	spin_unlock_irq(&dev_priv->uncore.lock);
diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index ac2082b022f3..c5dae9603a99 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -617,6 +617,12 @@ struct intel_uncore {
 		int id;
 		unsigned wake_count;
 		struct timer_list timer;
+		u32 reg_set;
+		u32 val_set;
+		u32 val_clear;
+		u32 reg_ack;
+		u32 reg_post;
+		u32 val_reset;
 	} fw_domain[FW_DOMAIN_ID_COUNT];
 #define FORCEWAKE_RENDER	(1 << FW_DOMAIN_ID_RENDER)
 #define FORCEWAKE_BLITTER	(1 << FW_DOMAIN_ID_BLITTER)
@@ -2557,6 +2563,7 @@ extern void intel_uncore_init(struct drm_device *dev);
 extern void intel_uncore_check_errors(struct drm_device *dev);
 extern void intel_uncore_fini(struct drm_device *dev);
 extern void intel_uncore_forcewake_reset(struct drm_device *dev, bool restore);
+const char *intel_uncore_forcewake_domain_to_str(const int domain_id);
 
 void
 i915_enable_pipestat(struct drm_i915_private *dev_priv, enum pipe pipe,
diff --git a/drivers/gpu/drm/i915/intel_uncore.c b/drivers/gpu/drm/i915/intel_uncore.c
index b35f3a9e6869..9991fdf24a7b 100644
--- a/drivers/gpu/drm/i915/intel_uncore.c
+++ b/drivers/gpu/drm/i915/intel_uncore.c
@@ -42,6 +42,26 @@
 
 #define __raw_posting_read(dev_priv__, reg__) (void)__raw_i915_read32(dev_priv__, reg__)
 
+static const char * const forcewake_domain_names[] = {
+	"render",
+	"blitter",
+	"media",
+};
+
+const char *
+intel_uncore_forcewake_domain_to_str(const int id)
+{
+	BUILD_BUG_ON((sizeof(forcewake_domain_names)/sizeof(const char *)) !=
+		     FW_DOMAIN_ID_COUNT);
+
+	if (id >= 0 && id < FW_DOMAIN_ID_COUNT)
+		return forcewake_domain_names[id];
+
+	WARN_ON(id);
+
+	return "unknown";
+}
+
 static void
 assert_device_not_suspended(struct drm_i915_private *dev_priv)
 {
@@ -49,73 +69,125 @@ assert_device_not_suspended(struct drm_i915_private *dev_priv)
 		  "Device suspended\n");
 }
 
-static void __gen6_gt_wait_for_thread_c0(struct drm_i915_private *dev_priv)
+static inline void
+fw_domain_reset(const struct intel_uncore_forcewake_domain *d)
 {
-	/* w/a for a sporadic read returning 0 by waiting for the GT
-	 * thread to wake up.
-	 */
-	if (wait_for_atomic_us((__raw_i915_read32(dev_priv, GEN6_GT_THREAD_STATUS_REG) &
-				GEN6_GT_THREAD_STATUS_CORE_MASK) == 0, 500))
-		DRM_ERROR("GT thread status wait timed out\n");
+	__raw_i915_write32(d->i915, d->reg_set, d->val_reset);
 }
 
-static void __gen6_gt_force_wake_reset(struct drm_i915_private *dev_priv)
+static inline void
+fw_domain_arm_timer(struct intel_uncore_forcewake_domain *d)
 {
-	__raw_i915_write32(dev_priv, FORCEWAKE, 0);
-	/* something from same cacheline, but !FORCEWAKE */
-	__raw_posting_read(dev_priv, ECOBUS);
+	mod_timer_pinned(&d->timer, jiffies + 1);
 }
 
-static void __gen6_gt_force_wake_get(struct drm_i915_private *dev_priv,
-				     int fw_engine)
+static inline void
+fw_domain_wait_ack_clear(const struct intel_uncore_forcewake_domain *d)
 {
-	if (wait_for_atomic((__raw_i915_read32(dev_priv, FORCEWAKE_ACK) & 1) == 0,
+	if (wait_for_atomic((__raw_i915_read32(d->i915, d->reg_ack) &
+			     FORCEWAKE_KERNEL) == 0,
 			    FORCEWAKE_ACK_TIMEOUT_MS))
-		DRM_ERROR("Timed out waiting for forcewake old ack to clear.\n");
+		DRM_ERROR("%s: timed out waiting for forcewake ack to clear.\n",
+			  intel_uncore_forcewake_domain_to_str(d->id));
+}
 
-	__raw_i915_write32(dev_priv, FORCEWAKE, 1);
-	/* something from same cacheline, but !FORCEWAKE */
-	__raw_posting_read(dev_priv, ECOBUS);
+static inline void
+fw_domain_get(const struct intel_uncore_forcewake_domain *d)
+{
+	__raw_i915_write32(d->i915, d->reg_set, d->val_set);
+}
 
-	if (wait_for_atomic((__raw_i915_read32(dev_priv, FORCEWAKE_ACK) & 1),
+static inline void
+fw_domain_wait_ack(const struct intel_uncore_forcewake_domain *d)
+{
+	if (wait_for_atomic((__raw_i915_read32(d->i915, d->reg_ack) &
+			     FORCEWAKE_KERNEL),
 			    FORCEWAKE_ACK_TIMEOUT_MS))
-		DRM_ERROR("Timed out waiting for forcewake to ack request.\n");
+		DRM_ERROR("%s: timed out waiting for forcewake ack request.\n",
+			  intel_uncore_forcewake_domain_to_str(d->id));
+}
 
-	/* WaRsForcewakeWaitTC0:snb */
-	__gen6_gt_wait_for_thread_c0(dev_priv);
+static inline void
+fw_domain_put(const struct intel_uncore_forcewake_domain *d)
+{
+	__raw_i915_write32(d->i915, d->reg_set, d->val_clear);
 }
 
-static void __gen7_gt_force_wake_mt_reset(struct drm_i915_private *dev_priv)
+static inline void
+fw_domain_posting_read(const struct intel_uncore_forcewake_domain *d)
 {
-	__raw_i915_write32(dev_priv, FORCEWAKE_MT, _MASKED_BIT_DISABLE(0xffff));
-	/* something from same cacheline, but !FORCEWAKE_MT */
-	__raw_posting_read(dev_priv, ECOBUS);
+	/* something from same cacheline, but not from the set register */
+	if (d->reg_post)
+		__raw_posting_read(d->i915, d->reg_post);
 }
 
-static void __gen7_gt_force_wake_mt_get(struct drm_i915_private *dev_priv,
-					int fw_engine)
+static void
+fw_domains_get(struct drm_i915_private *dev_priv, int fw_domains)
 {
-	u32 forcewake_ack;
+	struct intel_uncore_forcewake_domain *d;
+	int id;
 
-	if (IS_HASWELL(dev_priv->dev) || IS_BROADWELL(dev_priv->dev))
-		forcewake_ack = FORCEWAKE_ACK_HSW;
-	else
-		forcewake_ack = FORCEWAKE_MT_ACK;
+	for_each_fw_domain_mask(d, fw_domains, dev_priv, id) {
+		fw_domain_wait_ack_clear(d);
+		fw_domain_get(d);
+		fw_domain_posting_read(d);
+		fw_domain_wait_ack(d);
+	}
+}
 
-	if (wait_for_atomic((__raw_i915_read32(dev_priv, forcewake_ack) & FORCEWAKE_KERNEL) == 0,
-			    FORCEWAKE_ACK_TIMEOUT_MS))
-		DRM_ERROR("Timed out waiting for forcewake old ack to clear.\n");
+static void
+fw_domains_put(struct drm_i915_private *dev_priv, int fw_domains)
+{
+	struct intel_uncore_forcewake_domain *d;
+	int id;
 
-	__raw_i915_write32(dev_priv, FORCEWAKE_MT,
-			   _MASKED_BIT_ENABLE(FORCEWAKE_KERNEL));
-	/* something from same cacheline, but !FORCEWAKE_MT */
-	__raw_posting_read(dev_priv, ECOBUS);
+	for_each_fw_domain_mask(d, fw_domains, dev_priv, id) {
+		fw_domain_put(d);
+		fw_domain_posting_read(d);
+	}
+}
 
-	if (wait_for_atomic((__raw_i915_read32(dev_priv, forcewake_ack) & FORCEWAKE_KERNEL),
-			    FORCEWAKE_ACK_TIMEOUT_MS))
-		DRM_ERROR("Timed out waiting for forcewake to ack request.\n");
+static void
+fw_domains_posting_read(struct drm_i915_private *dev_priv)
+{
+	struct intel_uncore_forcewake_domain *d;
+	int id;
+
+	/* No need to do for all, just do for first found */
+	for_each_fw_domain(d, dev_priv, id) {
+		fw_domain_posting_read(d);
+		break;
+	}
+}
+
+static void
+fw_domains_reset(struct drm_i915_private *dev_priv, const unsigned fw_domains)
+{
+	struct intel_uncore_forcewake_domain *d;
+	int id;
+
+	for_each_fw_domain_mask(d, fw_domains, dev_priv, id)
+		fw_domain_reset(d);
+
+	fw_domains_posting_read(dev_priv);
+}
+
+static void __gen6_gt_wait_for_thread_c0(struct drm_i915_private *dev_priv)
+{
+	/* w/a for a sporadic read returning 0 by waiting for the GT
+	 * thread to wake up.
+	 */
+	if (wait_for_atomic_us((__raw_i915_read32(dev_priv, GEN6_GT_THREAD_STATUS_REG) &
+				GEN6_GT_THREAD_STATUS_CORE_MASK) == 0, 500))
+		DRM_ERROR("GT thread status wait timed out\n");
+}
+
+static void fw_domains_get_with_thread_status(struct drm_i915_private *dev_priv,
+					      int fw_domains)
+{
+	fw_domains_get(dev_priv, fw_domains);
 
-	/* WaRsForcewakeWaitTC0:ivb,hsw */
+	/* WaRsForcewakeWaitTC0:snb,ivb,hsw,bdw,vlv */
 	__gen6_gt_wait_for_thread_c0(dev_priv);
 }
 
@@ -128,27 +200,13 @@ static void gen6_gt_check_fifodbg(struct drm_i915_private *dev_priv)
 		__raw_i915_write32(dev_priv, GTFIFODBG, gtfifodbg);
 }
 
-static void __gen6_gt_force_wake_put(struct drm_i915_private *dev_priv,
-				     int fw_engine)
+static void fw_domains_put_with_fifo(struct drm_i915_private *dev_priv,
+				     int fw_domains)
 {
-	__raw_i915_write32(dev_priv, FORCEWAKE, 0);
-	/* something from same cacheline, but !FORCEWAKE */
-	__raw_posting_read(dev_priv, ECOBUS);
+	fw_domains_put(dev_priv, fw_domains);
 	gen6_gt_check_fifodbg(dev_priv);
 }
 
-static void __gen7_gt_force_wake_mt_put(struct drm_i915_private *dev_priv,
-					int fw_engine)
-{
-	__raw_i915_write32(dev_priv, FORCEWAKE_MT,
-			   _MASKED_BIT_DISABLE(FORCEWAKE_KERNEL));
-	/* something from same cacheline, but !FORCEWAKE_MT */
-	__raw_posting_read(dev_priv, ECOBUS);
-
-	if (IS_GEN7(dev_priv->dev))
-		gen6_gt_check_fifodbg(dev_priv);
-}
-
 static int __gen6_gt_wait_for_fifo(struct drm_i915_private *dev_priv)
 {
 	int ret = 0;
@@ -176,165 +234,16 @@ static int __gen6_gt_wait_for_fifo(struct drm_i915_private *dev_priv)
 	return ret;
 }
 
-static void vlv_force_wake_reset(struct drm_i915_private *dev_priv)
-{
-	__raw_i915_write32(dev_priv, FORCEWAKE_VLV,
-			   _MASKED_BIT_DISABLE(0xffff));
-	__raw_i915_write32(dev_priv, FORCEWAKE_MEDIA_VLV,
-			   _MASKED_BIT_DISABLE(0xffff));
-	/* something from same cacheline, but !FORCEWAKE_VLV */
-	__raw_posting_read(dev_priv, FORCEWAKE_ACK_VLV);
-}
-
-static void __vlv_force_wake_get(struct drm_i915_private *dev_priv,
-				 int fw_engine)
-{
-	/* Check for Render Engine */
-	if (FORCEWAKE_RENDER & fw_engine) {
-		if (wait_for_atomic((__raw_i915_read32(dev_priv,
-						FORCEWAKE_ACK_VLV) &
-						FORCEWAKE_KERNEL) == 0,
-					FORCEWAKE_ACK_TIMEOUT_MS))
-			DRM_ERROR("Timed out: Render forcewake old ack to clear.\n");
-
-		__raw_i915_write32(dev_priv, FORCEWAKE_VLV,
-				   _MASKED_BIT_ENABLE(FORCEWAKE_KERNEL));
-
-		if (wait_for_atomic((__raw_i915_read32(dev_priv,
-						FORCEWAKE_ACK_VLV) &
-						FORCEWAKE_KERNEL),
-					FORCEWAKE_ACK_TIMEOUT_MS))
-			DRM_ERROR("Timed out: waiting for Render to ack.\n");
-	}
-
-	/* Check for Media Engine */
-	if (FORCEWAKE_MEDIA & fw_engine) {
-		if (wait_for_atomic((__raw_i915_read32(dev_priv,
-						FORCEWAKE_ACK_MEDIA_VLV) &
-						FORCEWAKE_KERNEL) == 0,
-					FORCEWAKE_ACK_TIMEOUT_MS))
-			DRM_ERROR("Timed out: Media forcewake old ack to clear.\n");
-
-		__raw_i915_write32(dev_priv, FORCEWAKE_MEDIA_VLV,
-				   _MASKED_BIT_ENABLE(FORCEWAKE_KERNEL));
-
-		if (wait_for_atomic((__raw_i915_read32(dev_priv,
-						FORCEWAKE_ACK_MEDIA_VLV) &
-						FORCEWAKE_KERNEL),
-					FORCEWAKE_ACK_TIMEOUT_MS))
-			DRM_ERROR("Timed out: waiting for media to ack.\n");
-	}
-}
-
 static void __vlv_force_wake_put(struct drm_i915_private *dev_priv,
 				 int fw_engine)
 {
-	/* Check for Render Engine */
-	if (FORCEWAKE_RENDER & fw_engine)
-		__raw_i915_write32(dev_priv, FORCEWAKE_VLV,
-					_MASKED_BIT_DISABLE(FORCEWAKE_KERNEL));
-
+	fw_domains_put(dev_priv, fw_engine);
+	fw_domains_posting_read(dev_priv);
 
-	/* Check for Media Engine */
-	if (FORCEWAKE_MEDIA & fw_engine)
-		__raw_i915_write32(dev_priv, FORCEWAKE_MEDIA_VLV,
-				_MASKED_BIT_DISABLE(FORCEWAKE_KERNEL));
-
-	/* something from same cacheline, but !FORCEWAKE_VLV */
-	__raw_posting_read(dev_priv, FORCEWAKE_ACK_VLV);
 	if (!IS_CHERRYVIEW(dev_priv->dev))
 		gen6_gt_check_fifodbg(dev_priv);
 }
 
-static void __gen9_gt_force_wake_mt_reset(struct drm_i915_private *dev_priv)
-{
-	__raw_i915_write32(dev_priv, FORCEWAKE_RENDER_GEN9,
-			_MASKED_BIT_DISABLE(0xffff));
-
-	__raw_i915_write32(dev_priv, FORCEWAKE_MEDIA_GEN9,
-			_MASKED_BIT_DISABLE(0xffff));
-
-	__raw_i915_write32(dev_priv, FORCEWAKE_BLITTER_GEN9,
-			_MASKED_BIT_DISABLE(0xffff));
-}
-
-static void
-__gen9_force_wake_get(struct drm_i915_private *dev_priv, int fw_engine)
-{
-	/* Check for Render Engine */
-	if (FORCEWAKE_RENDER & fw_engine) {
-		if (wait_for_atomic((__raw_i915_read32(dev_priv,
-						FORCEWAKE_ACK_RENDER_GEN9) &
-						FORCEWAKE_KERNEL) == 0,
-					FORCEWAKE_ACK_TIMEOUT_MS))
-			DRM_ERROR("Timed out: Render forcewake old ack to clear.\n");
-
-		__raw_i915_write32(dev_priv, FORCEWAKE_RENDER_GEN9,
-				   _MASKED_BIT_ENABLE(FORCEWAKE_KERNEL));
-
-		if (wait_for_atomic((__raw_i915_read32(dev_priv,
-						FORCEWAKE_ACK_RENDER_GEN9) &
-						FORCEWAKE_KERNEL),
-					FORCEWAKE_ACK_TIMEOUT_MS))
-			DRM_ERROR("Timed out: waiting for Render to ack.\n");
-	}
-
-	/* Check for Media Engine */
-	if (FORCEWAKE_MEDIA & fw_engine) {
-		if (wait_for_atomic((__raw_i915_read32(dev_priv,
-						FORCEWAKE_ACK_MEDIA_GEN9) &
-						FORCEWAKE_KERNEL) == 0,
-					FORCEWAKE_ACK_TIMEOUT_MS))
-			DRM_ERROR("Timed out: Media forcewake old ack to clear.\n");
-
-		__raw_i915_write32(dev_priv, FORCEWAKE_MEDIA_GEN9,
-				   _MASKED_BIT_ENABLE(FORCEWAKE_KERNEL));
-
-		if (wait_for_atomic((__raw_i915_read32(dev_priv,
-						FORCEWAKE_ACK_MEDIA_GEN9) &
-						FORCEWAKE_KERNEL),
-					FORCEWAKE_ACK_TIMEOUT_MS))
-			DRM_ERROR("Timed out: waiting for Media to ack.\n");
-	}
-
-	/* Check for Blitter Engine */
-	if (FORCEWAKE_BLITTER & fw_engine) {
-		if (wait_for_atomic((__raw_i915_read32(dev_priv,
-						FORCEWAKE_ACK_BLITTER_GEN9) &
-						FORCEWAKE_KERNEL) == 0,
-					FORCEWAKE_ACK_TIMEOUT_MS))
-			DRM_ERROR("Timed out: Blitter forcewake old ack to clear.\n");
-
-		__raw_i915_write32(dev_priv, FORCEWAKE_BLITTER_GEN9,
-				   _MASKED_BIT_ENABLE(FORCEWAKE_KERNEL));
-
-		if (wait_for_atomic((__raw_i915_read32(dev_priv,
-						FORCEWAKE_ACK_BLITTER_GEN9) &
-						FORCEWAKE_KERNEL),
-					FORCEWAKE_ACK_TIMEOUT_MS))
-			DRM_ERROR("Timed out: waiting for Blitter to ack.\n");
-	}
-}
-
-static void
-__gen9_force_wake_put(struct drm_i915_private *dev_priv, int fw_engine)
-{
-	/* Check for Render Engine */
-	if (FORCEWAKE_RENDER & fw_engine)
-		__raw_i915_write32(dev_priv, FORCEWAKE_RENDER_GEN9,
-				_MASKED_BIT_DISABLE(FORCEWAKE_KERNEL));
-
-	/* Check for Media Engine */
-	if (FORCEWAKE_MEDIA & fw_engine)
-		__raw_i915_write32(dev_priv, FORCEWAKE_MEDIA_GEN9,
-				_MASKED_BIT_DISABLE(FORCEWAKE_KERNEL));
-
-	/* Check for Blitter Engine */
-	if (FORCEWAKE_BLITTER & fw_engine)
-		__raw_i915_write32(dev_priv, FORCEWAKE_BLITTER_GEN9,
-				_MASKED_BIT_DISABLE(FORCEWAKE_KERNEL));
-}
-
 static void gen6_force_wake_timer(unsigned long arg)
 {
 	struct intel_uncore_forcewake_domain *domain = (void *)arg;
@@ -402,16 +311,7 @@ void intel_uncore_forcewake_reset(struct drm_device *dev, bool restore)
 	if (fw)
 		dev_priv->uncore.funcs.force_wake_put(dev_priv, fw);
 
-	if (IS_VALLEYVIEW(dev))
-		vlv_force_wake_reset(dev_priv);
-	else if (IS_GEN6(dev) || IS_GEN7(dev))
-		__gen6_gt_force_wake_reset(dev_priv);
-
-	if (IS_IVYBRIDGE(dev) || IS_HASWELL(dev) || IS_BROADWELL(dev))
-		__gen7_gt_force_wake_mt_reset(dev_priv);
-
-	if (IS_GEN9(dev))
-		__gen9_gt_force_wake_mt_reset(dev_priv);
+	fw_domains_reset(dev_priv, FORCEWAKE_ALL);
 
 	if (restore) { /* If reset with a user forcewake, try to restore */
 		if (fw)
@@ -526,7 +426,7 @@ void gen6_gt_force_wake_put(struct drm_i915_private *dev_priv,
 			continue;
 
 		domain->wake_count++;
-		mod_timer_pinned(&domain->timer, jiffies + 1);
+		fw_domain_arm_timer(domain);
 	}
 
 	spin_unlock_irqrestore(&dev_priv->uncore.lock, irqflags);
@@ -535,12 +435,12 @@ void gen6_gt_force_wake_put(struct drm_i915_private *dev_priv,
 void assert_force_wake_inactive(struct drm_i915_private *dev_priv)
 {
 	struct intel_uncore_forcewake_domain *domain;
-	int i;
+	int id;
 
 	if (!dev_priv->uncore.funcs.force_wake_get)
 		return;
 
-	for_each_fw_domain(domain, dev_priv, i)
+	for_each_fw_domain(domain, dev_priv, id)
 		WARN_ON(domain->wake_count);
 }
 
@@ -708,20 +608,20 @@ static inline void __force_wake_get(struct drm_i915_private *dev_priv,
 				    unsigned fw_domains)
 {
 	struct intel_uncore_forcewake_domain *domain;
-	int i;
+	int id;
 
 	if (WARN_ON(!fw_domains))
 		return;
 
 	/* Ideally GCC would be constant-fold and eliminate this loop */
-	for_each_fw_domain_mask(domain, fw_domains, dev_priv, i) {
+	for_each_fw_domain_mask(domain, fw_domains, dev_priv, id) {
 		if (domain->wake_count) {
-			fw_domains &= ~(1 << i);
+			fw_domains &= ~(1 << id);
 			continue;
 		}
 
 		domain->wake_count++;
-		mod_timer_pinned(&domain->timer, jiffies + 1);
+		fw_domain_arm_timer(domain);
 	}
 
 	if (fw_domains)
@@ -1037,27 +937,78 @@ do { \
 	dev_priv->uncore.funcs.mmio_readq = x##_read64; \
 } while (0)
 
+
+static void fw_domain_init(struct drm_i915_private *dev_priv,
+			   u32 domain_id, u32 reg_set, u32 reg_ack)
+{
+	struct intel_uncore_forcewake_domain *d;
+
+	if (WARN_ON(domain_id >= FW_DOMAIN_ID_COUNT))
+		return;
+
+	d = &dev_priv->uncore.fw_domain[domain_id];
+
+	WARN_ON(d->wake_count);
+
+	d->wake_count = 0;
+	d->reg_set = reg_set;
+	d->reg_ack = reg_ack;
+
+	if (IS_GEN6(dev_priv)) {
+		d->val_reset = 0;
+		d->val_set = FORCEWAKE_KERNEL;
+		d->val_clear = 0;
+	} else {
+		d->val_reset = _MASKED_BIT_DISABLE(0xffff);
+		d->val_set = _MASKED_BIT_ENABLE(FORCEWAKE_KERNEL);
+		d->val_clear = _MASKED_BIT_DISABLE(FORCEWAKE_KERNEL);
+	}
+
+	if (IS_VALLEYVIEW(dev_priv))
+		d->reg_post = FORCEWAKE_ACK_VLV;
+	else if (IS_GEN6(dev_priv) || IS_GEN7(dev_priv) || IS_GEN8(dev_priv))
+		d->reg_post = ECOBUS;
+	else
+		d->reg_post = 0;
+
+	d->i915 = dev_priv;
+	d->id = domain_id;
+
+	setup_timer(&d->timer, gen6_force_wake_timer, (unsigned long)d);
+
+	dev_priv->uncore.fw_domains |= (1 << domain_id);
+}
+
 void intel_uncore_init(struct drm_device *dev)
 {
 	struct drm_i915_private *dev_priv = dev->dev_private;
-	struct intel_uncore_forcewake_domain *domain;
-	int i;
 
 	__intel_uncore_early_sanitize(dev, false);
 
 	if (IS_GEN9(dev)) {
-		dev_priv->uncore.funcs.force_wake_get = __gen9_force_wake_get;
-		dev_priv->uncore.funcs.force_wake_put = __gen9_force_wake_put;
-		dev_priv->uncore.fw_domains = FORCEWAKE_RENDER |
-			FORCEWAKE_BLITTER | FORCEWAKE_MEDIA;
+		dev_priv->uncore.funcs.force_wake_get = fw_domains_get;
+		dev_priv->uncore.funcs.force_wake_put = fw_domains_put;
+		fw_domain_init(dev_priv, FW_DOMAIN_ID_RENDER,
+			       FORCEWAKE_RENDER_GEN9,
+			       FORCEWAKE_ACK_RENDER_GEN9);
+		fw_domain_init(dev_priv, FW_DOMAIN_ID_BLITTER,
+			       FORCEWAKE_BLITTER_GEN9,
+			       FORCEWAKE_ACK_BLITTER_GEN9);
+		fw_domain_init(dev_priv, FW_DOMAIN_ID_MEDIA,
+			       FORCEWAKE_MEDIA_GEN9, FORCEWAKE_ACK_MEDIA_GEN9);
 	} else if (IS_VALLEYVIEW(dev)) {
-		dev_priv->uncore.funcs.force_wake_get = __vlv_force_wake_get;
+		dev_priv->uncore.funcs.force_wake_get = fw_domains_get;
 		dev_priv->uncore.funcs.force_wake_put = __vlv_force_wake_put;
-		dev_priv->uncore.fw_domains = FORCEWAKE_RENDER | FORCEWAKE_MEDIA;
+		fw_domain_init(dev_priv, FW_DOMAIN_ID_RENDER,
+			       FORCEWAKE_VLV, FORCEWAKE_ACK_VLV);
+		fw_domain_init(dev_priv, FW_DOMAIN_ID_MEDIA,
+			       FORCEWAKE_MEDIA_VLV, FORCEWAKE_ACK_MEDIA_VLV);
 	} else if (IS_HASWELL(dev) || IS_BROADWELL(dev)) {
-		dev_priv->uncore.funcs.force_wake_get = __gen7_gt_force_wake_mt_get;
-		dev_priv->uncore.funcs.force_wake_put = __gen7_gt_force_wake_mt_put;
-		dev_priv->uncore.fw_domains = FORCEWAKE_RENDER;
+		dev_priv->uncore.funcs.force_wake_get =
+			fw_domains_get_with_thread_status;
+		dev_priv->uncore.funcs.force_wake_put = fw_domains_put;
+		fw_domain_init(dev_priv, FW_DOMAIN_ID_RENDER,
+			       FORCEWAKE_MT, FORCEWAKE_ACK_HSW);
 	} else if (IS_IVYBRIDGE(dev)) {
 		u32 ecobus;
 
@@ -1070,40 +1021,32 @@ void intel_uncore_init(struct drm_device *dev)
 		 * (correctly) interpreted by the test below as MT
 		 * forcewake being disabled.
 		 */
+		dev_priv->uncore.funcs.force_wake_get =
+			fw_domains_get_with_thread_status;
+		dev_priv->uncore.funcs.force_wake_put =
+			fw_domains_put_with_fifo;
+
+		fw_domain_init(dev_priv, FW_DOMAIN_ID_RENDER,
+			       FORCEWAKE_MT, FORCEWAKE_MT_ACK);
 		mutex_lock(&dev->struct_mutex);
-		__gen7_gt_force_wake_mt_get(dev_priv, FORCEWAKE_ALL);
+		fw_domains_get_with_thread_status(dev_priv, FORCEWAKE_ALL);
 		ecobus = __raw_i915_read32(dev_priv, ECOBUS);
-		__gen7_gt_force_wake_mt_put(dev_priv, FORCEWAKE_ALL);
+		fw_domains_put_with_fifo(dev_priv, FORCEWAKE_ALL);
 		mutex_unlock(&dev->struct_mutex);
 
-		if (ecobus & FORCEWAKE_MT_ENABLE) {
-			dev_priv->uncore.funcs.force_wake_get =
-				__gen7_gt_force_wake_mt_get;
-			dev_priv->uncore.funcs.force_wake_put =
-				__gen7_gt_force_wake_mt_put;
-		} else {
+		if (!(ecobus & FORCEWAKE_MT_ENABLE)) {
 			DRM_INFO("No MT forcewake available on Ivybridge, this can result in issues\n");
 			DRM_INFO("when using vblank-synced partial screen updates.\n");
-			dev_priv->uncore.funcs.force_wake_get =
-				__gen6_gt_force_wake_get;
-			dev_priv->uncore.funcs.force_wake_put =
-				__gen6_gt_force_wake_put;
+			fw_domain_init(dev_priv, FW_DOMAIN_ID_RENDER,
+				       FORCEWAKE, FORCEWAKE_ACK);
 		}
-		dev_priv->uncore.fw_domains = FORCEWAKE_RENDER;
 	} else if (IS_GEN6(dev)) {
 		dev_priv->uncore.funcs.force_wake_get =
-			__gen6_gt_force_wake_get;
+			fw_domains_get_with_thread_status;
 		dev_priv->uncore.funcs.force_wake_put =
-			__gen6_gt_force_wake_put;
-		dev_priv->uncore.fw_domains = FORCEWAKE_RENDER;
-	}
-
-	for_each_fw_domain(domain, dev_priv, i) {
-		domain->i915 = dev_priv;
-		domain->id = i;
-
-		setup_timer(&domain->timer, gen6_force_wake_timer,
-			    (unsigned long)domain);
+			fw_domains_put_with_fifo;
+		fw_domain_init(dev_priv, FW_DOMAIN_ID_RENDER,
+			       FORCEWAKE, FORCEWAKE_ACK);
 	}
 
 	switch (INTEL_INFO(dev)->gen) {
-- 
cgit v1.2.3


From 756c349dc947e5d8e0938f2af63c3a6dec038261 Mon Sep 17 00:00:00 2001
From: Mika Kuoppala <mika.kuoppala@linux.intel.com>
Date: Fri, 16 Jan 2015 11:34:39 +0200
Subject: drm/i915: Make vlv and chv forcewake put generic.

These two were using a fw dance logic where posting read was done
after both domain bit were set. When in other gens, the posting
read is done immediately after setting the forcewake bit for each
domain.

Now bring these in line with other gens.

Signed-off-by: Mika Kuoppala <mika.kuoppala@intel.com>
Reviewed-by: Deepak S <deepak.s@linux.intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
---
 drivers/gpu/drm/i915/intel_uncore.c | 16 +++++-----------
 1 file changed, 5 insertions(+), 11 deletions(-)

(limited to 'drivers/gpu/drm/i915/intel_uncore.c')

diff --git a/drivers/gpu/drm/i915/intel_uncore.c b/drivers/gpu/drm/i915/intel_uncore.c
index 9991fdf24a7b..c08a6f9200ff 100644
--- a/drivers/gpu/drm/i915/intel_uncore.c
+++ b/drivers/gpu/drm/i915/intel_uncore.c
@@ -234,16 +234,6 @@ static int __gen6_gt_wait_for_fifo(struct drm_i915_private *dev_priv)
 	return ret;
 }
 
-static void __vlv_force_wake_put(struct drm_i915_private *dev_priv,
-				 int fw_engine)
-{
-	fw_domains_put(dev_priv, fw_engine);
-	fw_domains_posting_read(dev_priv);
-
-	if (!IS_CHERRYVIEW(dev_priv->dev))
-		gen6_gt_check_fifodbg(dev_priv);
-}
-
 static void gen6_force_wake_timer(unsigned long arg)
 {
 	struct intel_uncore_forcewake_domain *domain = (void *)arg;
@@ -998,7 +988,11 @@ void intel_uncore_init(struct drm_device *dev)
 			       FORCEWAKE_MEDIA_GEN9, FORCEWAKE_ACK_MEDIA_GEN9);
 	} else if (IS_VALLEYVIEW(dev)) {
 		dev_priv->uncore.funcs.force_wake_get = fw_domains_get;
-		dev_priv->uncore.funcs.force_wake_put = __vlv_force_wake_put;
+		if (!IS_CHERRYVIEW(dev))
+			dev_priv->uncore.funcs.force_wake_put =
+				fw_domains_put_with_fifo;
+		else
+			dev_priv->uncore.funcs.force_wake_put = fw_domains_put;
 		fw_domain_init(dev_priv, FW_DOMAIN_ID_RENDER,
 			       FORCEWAKE_VLV, FORCEWAKE_ACK_VLV);
 		fw_domain_init(dev_priv, FW_DOMAIN_ID_MEDIA,
-- 
cgit v1.2.3


From 59bad947180ddc22665e8b2b5f8f9ef65e8aab7f Mon Sep 17 00:00:00 2001
From: Mika Kuoppala <mika.kuoppala@linux.intel.com>
Date: Fri, 16 Jan 2015 11:34:40 +0200
Subject: drm/i915: Rename the forcewake get/put functions

We have multiple forcewake domains now on recent gens. Change the
function naming to reflect this.

v2: More verbose names (Chris)
v3: Rebase
v4: Rebase
v5: Add documentation for forcewake_get/put

Signed-off-by: Mika Kuoppala <mika.kuoppala@intel.com>
Reviewed-by: Deepak S <deepak.s@linux.intel.com> (v2)
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
---
 drivers/gpu/drm/i915/i915_debugfs.c     |  8 +++---
 drivers/gpu/drm/i915/i915_drv.c         |  2 +-
 drivers/gpu/drm/i915/i915_drv.h         | 15 ++++-------
 drivers/gpu/drm/i915/intel_display.c    |  4 +--
 drivers/gpu/drm/i915/intel_fbc.c        |  4 +--
 drivers/gpu/drm/i915/intel_lrc.c        |  4 +--
 drivers/gpu/drm/i915/intel_pm.c         | 25 +++++++++----------
 drivers/gpu/drm/i915/intel_ringbuffer.c |  4 +--
 drivers/gpu/drm/i915/intel_uncore.c     | 44 +++++++++++++++++++++------------
 9 files changed, 58 insertions(+), 52 deletions(-)

(limited to 'drivers/gpu/drm/i915/intel_uncore.c')

diff --git a/drivers/gpu/drm/i915/i915_debugfs.c b/drivers/gpu/drm/i915/i915_debugfs.c
index db7fcf385cd8..c39f9c560a85 100644
--- a/drivers/gpu/drm/i915/i915_debugfs.c
+++ b/drivers/gpu/drm/i915/i915_debugfs.c
@@ -1105,7 +1105,7 @@ static int i915_frequency_info(struct seq_file *m, void *unused)
 		if (ret)
 			goto out;
 
-		gen6_gt_force_wake_get(dev_priv, FORCEWAKE_ALL);
+		intel_uncore_forcewake_get(dev_priv, FORCEWAKE_ALL);
 
 		reqf = I915_READ(GEN6_RPNSWREQ);
 		reqf &= ~GEN6_TURBO_DISABLE;
@@ -1132,7 +1132,7 @@ static int i915_frequency_info(struct seq_file *m, void *unused)
 			cagf = (rpstat & GEN6_CAGF_MASK) >> GEN6_CAGF_SHIFT;
 		cagf *= GT_FREQUENCY_MULTIPLIER;
 
-		gen6_gt_force_wake_put(dev_priv, FORCEWAKE_ALL);
+		intel_uncore_forcewake_put(dev_priv, FORCEWAKE_ALL);
 		mutex_unlock(&dev->struct_mutex);
 
 		if (IS_GEN6(dev) || IS_GEN7(dev)) {
@@ -4322,7 +4322,7 @@ static int i915_forcewake_open(struct inode *inode, struct file *file)
 		return 0;
 
 	intel_runtime_pm_get(dev_priv);
-	gen6_gt_force_wake_get(dev_priv, FORCEWAKE_ALL);
+	intel_uncore_forcewake_get(dev_priv, FORCEWAKE_ALL);
 
 	return 0;
 }
@@ -4335,7 +4335,7 @@ static int i915_forcewake_release(struct inode *inode, struct file *file)
 	if (INTEL_INFO(dev)->gen < 6)
 		return 0;
 
-	gen6_gt_force_wake_put(dev_priv, FORCEWAKE_ALL);
+	intel_uncore_forcewake_put(dev_priv, FORCEWAKE_ALL);
 	intel_runtime_pm_put(dev_priv);
 
 	return 0;
diff --git a/drivers/gpu/drm/i915/i915_drv.c b/drivers/gpu/drm/i915/i915_drv.c
index 55a3fef075ae..6484229dd10d 100644
--- a/drivers/gpu/drm/i915/i915_drv.c
+++ b/drivers/gpu/drm/i915/i915_drv.c
@@ -1431,7 +1431,7 @@ static int intel_runtime_suspend(struct device *device)
 		intel_opregion_notify_adapter(dev, PCI_D3hot);
 	}
 
-	assert_force_wake_inactive(dev_priv);
+	assert_forcewakes_inactive(dev_priv);
 
 	DRM_DEBUG_KMS("Device suspended\n");
 	return 0;
diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index c5dae9603a99..0575c529a9a5 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -2564,6 +2564,11 @@ extern void intel_uncore_check_errors(struct drm_device *dev);
 extern void intel_uncore_fini(struct drm_device *dev);
 extern void intel_uncore_forcewake_reset(struct drm_device *dev, bool restore);
 const char *intel_uncore_forcewake_domain_to_str(const int domain_id);
+void intel_uncore_forcewake_get(struct drm_i915_private *dev_priv,
+				unsigned fw_domains);
+void intel_uncore_forcewake_put(struct drm_i915_private *dev_priv,
+				unsigned fw_domains);
+void assert_forcewakes_inactive(struct drm_i915_private *dev_priv);
 
 void
 i915_enable_pipestat(struct drm_i915_private *dev_priv, enum pipe pipe,
@@ -3195,16 +3200,6 @@ extern void intel_display_print_error_state(struct drm_i915_error_state_buf *e,
 					    struct drm_device *dev,
 					    struct intel_display_error_state *error);
 
-/* On SNB platform, before reading ring registers forcewake bit
- * must be set to prevent GT core from power down and stale values being
- * returned.
- */
-void gen6_gt_force_wake_get(struct drm_i915_private *dev_priv,
-			    unsigned fw_domains);
-void gen6_gt_force_wake_put(struct drm_i915_private *dev_priv,
-			    unsigned fw_domains);
-void assert_force_wake_inactive(struct drm_i915_private *dev_priv);
-
 int sandybridge_pcode_read(struct drm_i915_private *dev_priv, u32 mbox, u32 *val);
 int sandybridge_pcode_write(struct drm_i915_private *dev_priv, u32 mbox, u32 val);
 
diff --git a/drivers/gpu/drm/i915/intel_display.c b/drivers/gpu/drm/i915/intel_display.c
index 61b123f1a652..8c5dfec98de1 100644
--- a/drivers/gpu/drm/i915/intel_display.c
+++ b/drivers/gpu/drm/i915/intel_display.c
@@ -7871,7 +7871,7 @@ static void hsw_restore_lcpll(struct drm_i915_private *dev_priv)
 	 * Make sure we're not on PC8 state before disabling PC8, otherwise
 	 * we'll hang the machine. To prevent PC8 state, just enable force_wake.
 	 */
-	gen6_gt_force_wake_get(dev_priv, FORCEWAKE_ALL);
+	intel_uncore_forcewake_get(dev_priv, FORCEWAKE_ALL);
 
 	if (val & LCPLL_POWER_DOWN_ALLOW) {
 		val &= ~LCPLL_POWER_DOWN_ALLOW;
@@ -7901,7 +7901,7 @@ static void hsw_restore_lcpll(struct drm_i915_private *dev_priv)
 			DRM_ERROR("Switching back to LCPLL failed\n");
 	}
 
-	gen6_gt_force_wake_put(dev_priv, FORCEWAKE_ALL);
+	intel_uncore_forcewake_put(dev_priv, FORCEWAKE_ALL);
 }
 
 /*
diff --git a/drivers/gpu/drm/i915/intel_fbc.c b/drivers/gpu/drm/i915/intel_fbc.c
index 5b1d7c4cfae6..ed9a012d58ae 100644
--- a/drivers/gpu/drm/i915/intel_fbc.c
+++ b/drivers/gpu/drm/i915/intel_fbc.c
@@ -182,7 +182,7 @@ static void snb_fbc_blit_update(struct drm_device *dev)
 
 	/* Blitter is part of Media powerwell on VLV. No impact of
 	 * his param in other platforms for now */
-	gen6_gt_force_wake_get(dev_priv, FORCEWAKE_MEDIA);
+	intel_uncore_forcewake_get(dev_priv, FORCEWAKE_MEDIA);
 
 	blt_ecoskpd = I915_READ(GEN6_BLITTER_ECOSKPD);
 	blt_ecoskpd |= GEN6_BLITTER_FBC_NOTIFY <<
@@ -195,7 +195,7 @@ static void snb_fbc_blit_update(struct drm_device *dev)
 	I915_WRITE(GEN6_BLITTER_ECOSKPD, blt_ecoskpd);
 	POSTING_READ(GEN6_BLITTER_ECOSKPD);
 
-	gen6_gt_force_wake_put(dev_priv, FORCEWAKE_MEDIA);
+	intel_uncore_forcewake_put(dev_priv, FORCEWAKE_MEDIA);
 }
 
 static void ilk_fbc_enable(struct drm_crtc *crtc)
diff --git a/drivers/gpu/drm/i915/intel_lrc.c b/drivers/gpu/drm/i915/intel_lrc.c
index 49c7d862f8a5..cbfdbddf2eeb 100644
--- a/drivers/gpu/drm/i915/intel_lrc.c
+++ b/drivers/gpu/drm/i915/intel_lrc.c
@@ -296,7 +296,7 @@ static void execlists_elsp_write(struct intel_engine_cs *ring,
 	desc[3] = (u32)(temp >> 32);
 	desc[2] = (u32)temp;
 
-	gen6_gt_force_wake_get(dev_priv, FORCEWAKE_ALL);
+	intel_uncore_forcewake_get(dev_priv, FORCEWAKE_ALL);
 	I915_WRITE(RING_ELSP(ring), desc[1]);
 	I915_WRITE(RING_ELSP(ring), desc[0]);
 	I915_WRITE(RING_ELSP(ring), desc[3]);
@@ -306,7 +306,7 @@ static void execlists_elsp_write(struct intel_engine_cs *ring,
 
 	/* ELSP is a wo register, so use another nearby reg for posting instead */
 	POSTING_READ(RING_EXECLIST_STATUS(ring));
-	gen6_gt_force_wake_put(dev_priv, FORCEWAKE_ALL);
+	intel_uncore_forcewake_put(dev_priv, FORCEWAKE_ALL);
 }
 
 static int execlists_update_context(struct drm_i915_gem_object *ctx_obj,
diff --git a/drivers/gpu/drm/i915/intel_pm.c b/drivers/gpu/drm/i915/intel_pm.c
index 292522260aa6..acfa362b042d 100644
--- a/drivers/gpu/drm/i915/intel_pm.c
+++ b/drivers/gpu/drm/i915/intel_pm.c
@@ -76,7 +76,6 @@ static void gen9_init_clock_gating(struct drm_device *dev)
 		   _MASKED_BIT_ENABLE(GEN8_4x4_STC_OPTIMIZATION_DISABLE));
 }
 
-
 static void i915_pineview_get_mem_freq(struct drm_device *dev)
 {
 	struct drm_i915_private *dev_priv = dev->dev_private;
@@ -3915,11 +3914,11 @@ static void valleyview_disable_rps(struct drm_device *dev)
 
 	/* we're doing forcewake before Disabling RC6,
 	 * This what the BIOS expects when going into suspend */
-	gen6_gt_force_wake_get(dev_priv, FORCEWAKE_ALL);
+	intel_uncore_forcewake_get(dev_priv, FORCEWAKE_ALL);
 
 	I915_WRITE(GEN6_RC_CONTROL, 0);
 
-	gen6_gt_force_wake_put(dev_priv, FORCEWAKE_ALL);
+	intel_uncore_forcewake_put(dev_priv, FORCEWAKE_ALL);
 }
 
 static void intel_print_rc6_info(struct drm_device *dev, u32 mode)
@@ -4037,7 +4036,7 @@ static void gen9_enable_rps(struct drm_device *dev)
 
 	/* 1b: Get forcewake during program sequence. Although the driver
 	 * hasn't enabled a state yet where we need forcewake, BIOS may have.*/
-	gen6_gt_force_wake_get(dev_priv, FORCEWAKE_ALL);
+	intel_uncore_forcewake_get(dev_priv, FORCEWAKE_ALL);
 
 	/* 2a: Disable RC states. */
 	I915_WRITE(GEN6_RC_CONTROL, 0);
@@ -4060,7 +4059,7 @@ static void gen9_enable_rps(struct drm_device *dev)
 				   GEN6_RC_CTL_EI_MODE(1) |
 				   rc6_mask);
 
-	gen6_gt_force_wake_put(dev_priv, FORCEWAKE_ALL);
+	intel_uncore_forcewake_put(dev_priv, FORCEWAKE_ALL);
 
 }
 
@@ -4076,7 +4075,7 @@ static void gen8_enable_rps(struct drm_device *dev)
 
 	/* 1c & 1d: Get forcewake during program sequence. Although the driver
 	 * hasn't enabled a state yet where we need forcewake, BIOS may have.*/
-	gen6_gt_force_wake_get(dev_priv, FORCEWAKE_ALL);
+	intel_uncore_forcewake_get(dev_priv, FORCEWAKE_ALL);
 
 	/* 2a: Disable RC states. */
 	I915_WRITE(GEN6_RC_CONTROL, 0);
@@ -4143,7 +4142,7 @@ static void gen8_enable_rps(struct drm_device *dev)
 	dev_priv->rps.power = HIGH_POWER; /* force a reset */
 	gen6_set_rps(dev_priv->dev, dev_priv->rps.min_freq_softlimit);
 
-	gen6_gt_force_wake_put(dev_priv, FORCEWAKE_ALL);
+	intel_uncore_forcewake_put(dev_priv, FORCEWAKE_ALL);
 }
 
 static void gen6_enable_rps(struct drm_device *dev)
@@ -4171,7 +4170,7 @@ static void gen6_enable_rps(struct drm_device *dev)
 		I915_WRITE(GTFIFODBG, gtfifodbg);
 	}
 
-	gen6_gt_force_wake_get(dev_priv, FORCEWAKE_ALL);
+	intel_uncore_forcewake_get(dev_priv, FORCEWAKE_ALL);
 
 	/* Initialize rps frequencies */
 	gen6_init_rps_frequencies(dev);
@@ -4251,7 +4250,7 @@ static void gen6_enable_rps(struct drm_device *dev)
 			DRM_ERROR("Couldn't fix incorrect rc6 voltage\n");
 	}
 
-	gen6_gt_force_wake_put(dev_priv, FORCEWAKE_ALL);
+	intel_uncore_forcewake_put(dev_priv, FORCEWAKE_ALL);
 }
 
 static void __gen6_update_ring_freq(struct drm_device *dev)
@@ -4710,7 +4709,7 @@ static void cherryview_enable_rps(struct drm_device *dev)
 
 	/* 1a & 1b: Get forcewake during program sequence. Although the driver
 	 * hasn't enabled a state yet where we need forcewake, BIOS may have.*/
-	gen6_gt_force_wake_get(dev_priv, FORCEWAKE_ALL);
+	intel_uncore_forcewake_get(dev_priv, FORCEWAKE_ALL);
 
 	/*  Disable RC states. */
 	I915_WRITE(GEN6_RC_CONTROL, 0);
@@ -4783,7 +4782,7 @@ static void cherryview_enable_rps(struct drm_device *dev)
 
 	valleyview_set_rps(dev_priv->dev, dev_priv->rps.efficient_freq);
 
-	gen6_gt_force_wake_put(dev_priv, FORCEWAKE_ALL);
+	intel_uncore_forcewake_put(dev_priv, FORCEWAKE_ALL);
 }
 
 static void valleyview_enable_rps(struct drm_device *dev)
@@ -4804,7 +4803,7 @@ static void valleyview_enable_rps(struct drm_device *dev)
 	}
 
 	/* If VLV, Forcewake all wells, else re-direct to regular path */
-	gen6_gt_force_wake_get(dev_priv, FORCEWAKE_ALL);
+	intel_uncore_forcewake_get(dev_priv, FORCEWAKE_ALL);
 
 	/*  Disable RC states. */
 	I915_WRITE(GEN6_RC_CONTROL, 0);
@@ -4867,7 +4866,7 @@ static void valleyview_enable_rps(struct drm_device *dev)
 
 	valleyview_set_rps(dev_priv->dev, dev_priv->rps.efficient_freq);
 
-	gen6_gt_force_wake_put(dev_priv, FORCEWAKE_ALL);
+	intel_uncore_forcewake_put(dev_priv, FORCEWAKE_ALL);
 }
 
 void ironlake_teardown_rc6(struct drm_device *dev)
diff --git a/drivers/gpu/drm/i915/intel_ringbuffer.c b/drivers/gpu/drm/i915/intel_ringbuffer.c
index b117717639fe..d7aa5c464d96 100644
--- a/drivers/gpu/drm/i915/intel_ringbuffer.c
+++ b/drivers/gpu/drm/i915/intel_ringbuffer.c
@@ -539,7 +539,7 @@ static int init_ring_common(struct intel_engine_cs *ring)
 	struct drm_i915_gem_object *obj = ringbuf->obj;
 	int ret = 0;
 
-	gen6_gt_force_wake_get(dev_priv, FORCEWAKE_ALL);
+	intel_uncore_forcewake_get(dev_priv, FORCEWAKE_ALL);
 
 	if (!stop_ring(ring)) {
 		/* G45 ring initialization often fails to reset head to zero */
@@ -611,7 +611,7 @@ static int init_ring_common(struct intel_engine_cs *ring)
 	memset(&ring->hangcheck, 0, sizeof(ring->hangcheck));
 
 out:
-	gen6_gt_force_wake_put(dev_priv, FORCEWAKE_ALL);
+	intel_uncore_forcewake_put(dev_priv, FORCEWAKE_ALL);
 
 	return ret;
 }
diff --git a/drivers/gpu/drm/i915/intel_uncore.c b/drivers/gpu/drm/i915/intel_uncore.c
index c08a6f9200ff..90c195108854 100644
--- a/drivers/gpu/drm/i915/intel_uncore.c
+++ b/drivers/gpu/drm/i915/intel_uncore.c
@@ -234,7 +234,7 @@ static int __gen6_gt_wait_for_fifo(struct drm_i915_private *dev_priv)
 	return ret;
 }
 
-static void gen6_force_wake_timer(unsigned long arg)
+static void intel_uncore_fw_release_timer(unsigned long arg)
 {
 	struct intel_uncore_forcewake_domain *domain = (void *)arg;
 	unsigned long irqflags;
@@ -270,7 +270,7 @@ void intel_uncore_forcewake_reset(struct drm_device *dev, bool restore)
 			if (del_timer_sync(&domain->timer) == 0)
 				continue;
 
-			gen6_force_wake_timer((unsigned long)domain);
+			intel_uncore_fw_release_timer((unsigned long)domain);
 		}
 
 		spin_lock_irqsave(&dev_priv->uncore.lock, irqflags);
@@ -314,7 +314,7 @@ void intel_uncore_forcewake_reset(struct drm_device *dev, bool restore)
 	}
 
 	if (!restore)
-		assert_force_wake_inactive(dev_priv);
+		assert_forcewakes_inactive(dev_priv);
 
 	spin_unlock_irqrestore(&dev_priv->uncore.lock, irqflags);
 }
@@ -358,14 +358,21 @@ void intel_uncore_sanitize(struct drm_device *dev)
 	intel_disable_gt_powersave(dev);
 }
 
-/*
- * Generally this is called implicitly by the register read function. However,
- * if some sequence requires the GT to not power down then this function should
- * be called at the beginning of the sequence followed by a call to
- * gen6_gt_force_wake_put() at the end of the sequence.
+/**
+ * intel_uncore_forcewake_get - grab forcewake domain references
+ * @dev_priv: i915 device instance
+ * @fw_domains: forcewake domains to get reference on
+ *
+ * This function can be used get GT's forcewake domain references.
+ * Normal register access will handle the forcewake domains automatically.
+ * However if some sequence requires the GT to not power down a particular
+ * forcewake domains this function should be called at the beginning of the
+ * sequence. And subsequently the reference should be dropped by symmetric
+ * call to intel_unforce_forcewake_put(). Usually caller wants all the domains
+ * to be kept awake so the @fw_domains would be then FORCEWAKE_ALL.
  */
-void gen6_gt_force_wake_get(struct drm_i915_private *dev_priv,
-			    unsigned fw_domains)
+void intel_uncore_forcewake_get(struct drm_i915_private *dev_priv,
+				unsigned fw_domains)
 {
 	unsigned long irqflags;
 	struct intel_uncore_forcewake_domain *domain;
@@ -391,11 +398,16 @@ void gen6_gt_force_wake_get(struct drm_i915_private *dev_priv,
 	spin_unlock_irqrestore(&dev_priv->uncore.lock, irqflags);
 }
 
-/*
- * see gen6_gt_force_wake_get()
+/**
+ * intel_uncore_forcewake_put - release a forcewake domain reference
+ * @dev_priv: i915 device instance
+ * @fw_domains: forcewake domains to put references
+ *
+ * This function drops the device-level forcewakes for specified
+ * domains obtained by intel_uncore_forcewake_get().
  */
-void gen6_gt_force_wake_put(struct drm_i915_private *dev_priv,
-			    unsigned fw_domains)
+void intel_uncore_forcewake_put(struct drm_i915_private *dev_priv,
+				unsigned fw_domains)
 {
 	unsigned long irqflags;
 	struct intel_uncore_forcewake_domain *domain;
@@ -422,7 +434,7 @@ void gen6_gt_force_wake_put(struct drm_i915_private *dev_priv,
 	spin_unlock_irqrestore(&dev_priv->uncore.lock, irqflags);
 }
 
-void assert_force_wake_inactive(struct drm_i915_private *dev_priv)
+void assert_forcewakes_inactive(struct drm_i915_private *dev_priv)
 {
 	struct intel_uncore_forcewake_domain *domain;
 	int id;
@@ -964,7 +976,7 @@ static void fw_domain_init(struct drm_i915_private *dev_priv,
 	d->i915 = dev_priv;
 	d->id = domain_id;
 
-	setup_timer(&d->timer, gen6_force_wake_timer, (unsigned long)d);
+	setup_timer(&d->timer, intel_uncore_fw_release_timer, (unsigned long)d);
 
 	dev_priv->uncore.fw_domains |= (1 << domain_id);
 }
-- 
cgit v1.2.3


From 48c1026a9eab9f3b3e35484fb1d2ee26d7587b36 Mon Sep 17 00:00:00 2001
From: Mika Kuoppala <mika.kuoppala@linux.intel.com>
Date: Fri, 16 Jan 2015 11:34:41 +0200
Subject: drm/i915: Enum forcewake domains and domain identifiers

Make the domains and domain identifiers enums. To emphasize
the difference in order to avoid mistakes.

v2: s/fw_domain/forcewake_domain (Jani)
v3: rebase

Suggested-by: Daniel Vetter <daniel.vetter@ffwll.ch>
Signed-off-by: Mika Kuoppala <mika.kuoppala@intel.com>
Reviewed-by: Deepak S <deepak.s@linux.intel.com> (v1)
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
---
 drivers/gpu/drm/i915/i915_drv.h     | 45 ++++++++++++++++++-----------------
 drivers/gpu/drm/i915/intel_uncore.c | 47 ++++++++++++++++++++-----------------
 2 files changed, 49 insertions(+), 43 deletions(-)

(limited to 'drivers/gpu/drm/i915/intel_uncore.c')

diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index 0575c529a9a5..b2a7f7a115d2 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -575,11 +575,28 @@ struct drm_i915_display_funcs {
 	void (*enable_backlight)(struct intel_connector *connector);
 };
 
+enum forcewake_domain_id {
+	FW_DOMAIN_ID_RENDER = 0,
+	FW_DOMAIN_ID_BLITTER,
+	FW_DOMAIN_ID_MEDIA,
+
+	FW_DOMAIN_ID_COUNT
+};
+
+enum forcewake_domains {
+	FORCEWAKE_RENDER = (1 << FW_DOMAIN_ID_RENDER),
+	FORCEWAKE_BLITTER = (1 << FW_DOMAIN_ID_BLITTER),
+	FORCEWAKE_MEDIA	= (1 << FW_DOMAIN_ID_MEDIA),
+	FORCEWAKE_ALL = (FORCEWAKE_RENDER |
+			 FORCEWAKE_BLITTER |
+			 FORCEWAKE_MEDIA)
+};
+
 struct intel_uncore_funcs {
 	void (*force_wake_get)(struct drm_i915_private *dev_priv,
-							int fw_engine);
+							enum forcewake_domains domains);
 	void (*force_wake_put)(struct drm_i915_private *dev_priv,
-							int fw_engine);
+							enum forcewake_domains domains);
 
 	uint8_t  (*mmio_readb)(struct drm_i915_private *dev_priv, off_t offset, bool trace);
 	uint16_t (*mmio_readw)(struct drm_i915_private *dev_priv, off_t offset, bool trace);
@@ -596,25 +613,17 @@ struct intel_uncore_funcs {
 				uint64_t val, bool trace);
 };
 
-enum {
-	FW_DOMAIN_ID_RENDER = 0,
-	FW_DOMAIN_ID_BLITTER,
-	FW_DOMAIN_ID_MEDIA,
-
-	FW_DOMAIN_ID_COUNT
-};
-
 struct intel_uncore {
 	spinlock_t lock; /** lock is also taken in irq contexts. */
 
 	struct intel_uncore_funcs funcs;
 
 	unsigned fifo_count;
-	unsigned fw_domains;
+	enum forcewake_domains fw_domains;
 
 	struct intel_uncore_forcewake_domain {
 		struct drm_i915_private *i915;
-		int id;
+		enum forcewake_domain_id id;
 		unsigned wake_count;
 		struct timer_list timer;
 		u32 reg_set;
@@ -624,12 +633,6 @@ struct intel_uncore {
 		u32 reg_post;
 		u32 val_reset;
 	} fw_domain[FW_DOMAIN_ID_COUNT];
-#define FORCEWAKE_RENDER	(1 << FW_DOMAIN_ID_RENDER)
-#define FORCEWAKE_BLITTER	(1 << FW_DOMAIN_ID_BLITTER)
-#define FORCEWAKE_MEDIA		(1 << FW_DOMAIN_ID_MEDIA)
-#define FORCEWAKE_ALL		(FORCEWAKE_RENDER | \
-				 FORCEWAKE_BLITTER | \
-				 FORCEWAKE_MEDIA)
 };
 
 /* Iterate over initialised fw domains */
@@ -2563,11 +2566,11 @@ extern void intel_uncore_init(struct drm_device *dev);
 extern void intel_uncore_check_errors(struct drm_device *dev);
 extern void intel_uncore_fini(struct drm_device *dev);
 extern void intel_uncore_forcewake_reset(struct drm_device *dev, bool restore);
-const char *intel_uncore_forcewake_domain_to_str(const int domain_id);
+const char *intel_uncore_forcewake_domain_to_str(const enum forcewake_domain_id id);
 void intel_uncore_forcewake_get(struct drm_i915_private *dev_priv,
-				unsigned fw_domains);
+				enum forcewake_domains domains);
 void intel_uncore_forcewake_put(struct drm_i915_private *dev_priv,
-				unsigned fw_domains);
+				enum forcewake_domains domains);
 void assert_forcewakes_inactive(struct drm_i915_private *dev_priv);
 
 void
diff --git a/drivers/gpu/drm/i915/intel_uncore.c b/drivers/gpu/drm/i915/intel_uncore.c
index 90c195108854..b3951f288a90 100644
--- a/drivers/gpu/drm/i915/intel_uncore.c
+++ b/drivers/gpu/drm/i915/intel_uncore.c
@@ -49,7 +49,7 @@ static const char * const forcewake_domain_names[] = {
 };
 
 const char *
-intel_uncore_forcewake_domain_to_str(const int id)
+intel_uncore_forcewake_domain_to_str(const enum forcewake_domain_id id)
 {
 	BUILD_BUG_ON((sizeof(forcewake_domain_names)/sizeof(const char *)) !=
 		     FW_DOMAIN_ID_COUNT);
@@ -122,10 +122,10 @@ fw_domain_posting_read(const struct intel_uncore_forcewake_domain *d)
 }
 
 static void
-fw_domains_get(struct drm_i915_private *dev_priv, int fw_domains)
+fw_domains_get(struct drm_i915_private *dev_priv, enum forcewake_domains fw_domains)
 {
 	struct intel_uncore_forcewake_domain *d;
-	int id;
+	enum forcewake_domain_id id;
 
 	for_each_fw_domain_mask(d, fw_domains, dev_priv, id) {
 		fw_domain_wait_ack_clear(d);
@@ -136,10 +136,10 @@ fw_domains_get(struct drm_i915_private *dev_priv, int fw_domains)
 }
 
 static void
-fw_domains_put(struct drm_i915_private *dev_priv, int fw_domains)
+fw_domains_put(struct drm_i915_private *dev_priv, enum forcewake_domains fw_domains)
 {
 	struct intel_uncore_forcewake_domain *d;
-	int id;
+	enum forcewake_domain_id id;
 
 	for_each_fw_domain_mask(d, fw_domains, dev_priv, id) {
 		fw_domain_put(d);
@@ -151,7 +151,7 @@ static void
 fw_domains_posting_read(struct drm_i915_private *dev_priv)
 {
 	struct intel_uncore_forcewake_domain *d;
-	int id;
+	enum forcewake_domain_id id;
 
 	/* No need to do for all, just do for first found */
 	for_each_fw_domain(d, dev_priv, id) {
@@ -161,10 +161,10 @@ fw_domains_posting_read(struct drm_i915_private *dev_priv)
 }
 
 static void
-fw_domains_reset(struct drm_i915_private *dev_priv, const unsigned fw_domains)
+fw_domains_reset(struct drm_i915_private *dev_priv, enum forcewake_domains fw_domains)
 {
 	struct intel_uncore_forcewake_domain *d;
-	int id;
+	enum forcewake_domain_id id;
 
 	for_each_fw_domain_mask(d, fw_domains, dev_priv, id)
 		fw_domain_reset(d);
@@ -183,7 +183,7 @@ static void __gen6_gt_wait_for_thread_c0(struct drm_i915_private *dev_priv)
 }
 
 static void fw_domains_get_with_thread_status(struct drm_i915_private *dev_priv,
-					      int fw_domains)
+					      enum forcewake_domains fw_domains)
 {
 	fw_domains_get(dev_priv, fw_domains);
 
@@ -201,7 +201,7 @@ static void gen6_gt_check_fifodbg(struct drm_i915_private *dev_priv)
 }
 
 static void fw_domains_put_with_fifo(struct drm_i915_private *dev_priv,
-				     int fw_domains)
+				     enum forcewake_domains fw_domains)
 {
 	fw_domains_put(dev_priv, fw_domains);
 	gen6_gt_check_fifodbg(dev_priv);
@@ -255,9 +255,11 @@ static void intel_uncore_fw_release_timer(unsigned long arg)
 void intel_uncore_forcewake_reset(struct drm_device *dev, bool restore)
 {
 	struct drm_i915_private *dev_priv = dev->dev_private;
-	unsigned long irqflags, fw = 0;
+	unsigned long irqflags;
 	struct intel_uncore_forcewake_domain *domain;
-	int id, active_domains, retry_count = 100;
+	int retry_count = 100;
+	enum forcewake_domain_id id;
+	enum forcewake_domains fw = 0, active_domains;
 
 	/* Hold uncore.lock across reset to prevent any register access
 	 * with forcewake not set correctly. Wait until all pending
@@ -372,11 +374,11 @@ void intel_uncore_sanitize(struct drm_device *dev)
  * to be kept awake so the @fw_domains would be then FORCEWAKE_ALL.
  */
 void intel_uncore_forcewake_get(struct drm_i915_private *dev_priv,
-				unsigned fw_domains)
+				enum forcewake_domains fw_domains)
 {
 	unsigned long irqflags;
 	struct intel_uncore_forcewake_domain *domain;
-	int id;
+	enum forcewake_domain_id id;
 
 	if (!dev_priv->uncore.funcs.force_wake_get)
 		return;
@@ -407,11 +409,11 @@ void intel_uncore_forcewake_get(struct drm_i915_private *dev_priv,
  * domains obtained by intel_uncore_forcewake_get().
  */
 void intel_uncore_forcewake_put(struct drm_i915_private *dev_priv,
-				unsigned fw_domains)
+				enum forcewake_domains fw_domains)
 {
 	unsigned long irqflags;
 	struct intel_uncore_forcewake_domain *domain;
-	int id;
+	enum forcewake_domain_id id;
 
 	if (!dev_priv->uncore.funcs.force_wake_put)
 		return;
@@ -437,7 +439,7 @@ void intel_uncore_forcewake_put(struct drm_i915_private *dev_priv,
 void assert_forcewakes_inactive(struct drm_i915_private *dev_priv)
 {
 	struct intel_uncore_forcewake_domain *domain;
-	int id;
+	enum forcewake_domain_id id;
 
 	if (!dev_priv->uncore.funcs.force_wake_get)
 		return;
@@ -607,10 +609,10 @@ __gen2_read(64)
 	return val
 
 static inline void __force_wake_get(struct drm_i915_private *dev_priv,
-				    unsigned fw_domains)
+				    enum forcewake_domains fw_domains)
 {
 	struct intel_uncore_forcewake_domain *domain;
-	int id;
+	enum forcewake_domain_id id;
 
 	if (WARN_ON(!fw_domains))
 		return;
@@ -675,7 +677,7 @@ chv_read##x(struct drm_i915_private *dev_priv, off_t reg, bool trace) { \
 #define __gen9_read(x) \
 static u##x \
 gen9_read##x(struct drm_i915_private *dev_priv, off_t reg, bool trace) { \
-	unsigned fw_engine; \
+	enum forcewake_domains fw_engine; \
 	GEN6_READ_HEADER(x); \
 	if (!SKL_NEEDS_FORCE_WAKE((dev_priv), (reg)))	\
 		fw_engine = 0; \
@@ -875,7 +877,7 @@ static bool is_gen9_shadowed(struct drm_i915_private *dev_priv, u32 reg)
 static void \
 gen9_write##x(struct drm_i915_private *dev_priv, off_t reg, u##x val, \
 		bool trace) { \
-	unsigned fw_engine; \
+	enum forcewake_domains fw_engine; \
 	GEN6_WRITE_HEADER; \
 	if (!SKL_NEEDS_FORCE_WAKE((dev_priv), (reg)) ||	\
 	    is_gen9_shadowed(dev_priv, reg)) \
@@ -941,7 +943,8 @@ do { \
 
 
 static void fw_domain_init(struct drm_i915_private *dev_priv,
-			   u32 domain_id, u32 reg_set, u32 reg_ack)
+			   enum forcewake_domain_id domain_id,
+			   u32 reg_set, u32 reg_ack)
 {
 	struct intel_uncore_forcewake_domain *d;
 
-- 
cgit v1.2.3


From f9b3927afbb97649f5c89f0815820167b7c5cba8 Mon Sep 17 00:00:00 2001
From: Mika Kuoppala <mika.kuoppala@linux.intel.com>
Date: Wed, 28 Jan 2015 14:43:24 +0200
Subject: drm/i915: Do uncore early sanitize after domain init

intel_uncore_early_sanitize() will reset the forcewake registers. When
forcewake domains were introduced, the domain init was done after the
sanitization of the forcewake registers. And as the resetting of
registers use the domain accessors, we tried to reset the forcewake
registers with unitialized forcewake domains and failed.

Fix this by sanitizing after all the domains have been initialized. Do
per domain clearing of forcewake register on domain init so that
IVB can do early access to ECOBUS do determine the final configuration.

This regression was introduced in

commit 05a2fb157e44a53c79133805d30eaada43911941
Author: Mika Kuoppala <mika.kuoppala@linux.intel.com>
Date:   Mon Jan 19 16:20:43 2015 +0200

    drm/i915: Consolidate forcewake code

v2: Carve out ellc detect, fw_domain_reset for ivb/ecobus (Chris)

Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=88805
Cc: Chris Wilson <chris@chris-wilson.co.uk>
Reported-by: Olof Johansson <olof@lixom.net>
Tested-by: Darren Hart <dvhart@linux.intel.com> (v1)
Signed-off-by: Mika Kuoppala <mika.kuoppala@intel.com>
Reviewed-by: Chris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
---
 drivers/gpu/drm/i915/intel_uncore.c | 38 +++++++++++++++++++++++++++++--------
 1 file changed, 30 insertions(+), 8 deletions(-)

(limited to 'drivers/gpu/drm/i915/intel_uncore.c')

diff --git a/drivers/gpu/drm/i915/intel_uncore.c b/drivers/gpu/drm/i915/intel_uncore.c
index b3951f288a90..be2c7fcf5f1f 100644
--- a/drivers/gpu/drm/i915/intel_uncore.c
+++ b/drivers/gpu/drm/i915/intel_uncore.c
@@ -72,6 +72,7 @@ assert_device_not_suspended(struct drm_i915_private *dev_priv)
 static inline void
 fw_domain_reset(const struct intel_uncore_forcewake_domain *d)
 {
+	WARN_ON(d->reg_set == 0);
 	__raw_i915_write32(d->i915, d->reg_set, d->val_reset);
 }
 
@@ -166,6 +167,8 @@ fw_domains_reset(struct drm_i915_private *dev_priv, enum forcewake_domains fw_do
 	struct intel_uncore_forcewake_domain *d;
 	enum forcewake_domain_id id;
 
+	WARN_ON(dev_priv->uncore.fw_domains == 0);
+
 	for_each_fw_domain_mask(d, fw_domains, dev_priv, id)
 		fw_domain_reset(d);
 
@@ -321,14 +324,10 @@ void intel_uncore_forcewake_reset(struct drm_device *dev, bool restore)
 	spin_unlock_irqrestore(&dev_priv->uncore.lock, irqflags);
 }
 
-static void __intel_uncore_early_sanitize(struct drm_device *dev,
-					  bool restore_forcewake)
+static void intel_uncore_ellc_detect(struct drm_device *dev)
 {
 	struct drm_i915_private *dev_priv = dev->dev_private;
 
-	if (HAS_FPGA_DBG_UNCLAIMED(dev))
-		__raw_i915_write32(dev_priv, FPGA_DBG, FPGA_DBG_RM_NOCLAIM);
-
 	if ((IS_HASWELL(dev) || IS_BROADWELL(dev)) &&
 	    (__raw_i915_read32(dev_priv, HSW_EDRAM_PRESENT) == 1)) {
 		/* The docs do not explain exactly how the calculation can be
@@ -339,6 +338,15 @@ static void __intel_uncore_early_sanitize(struct drm_device *dev,
 		dev_priv->ellc_size = 128;
 		DRM_INFO("Found %zuMB of eLLC\n", dev_priv->ellc_size);
 	}
+}
+
+static void __intel_uncore_early_sanitize(struct drm_device *dev,
+					  bool restore_forcewake)
+{
+	struct drm_i915_private *dev_priv = dev->dev_private;
+
+	if (HAS_FPGA_DBG_UNCLAIMED(dev))
+		__raw_i915_write32(dev_priv, FPGA_DBG, FPGA_DBG_RM_NOCLAIM);
 
 	/* clear out old GT FIFO errors */
 	if (IS_GEN6(dev) || IS_GEN7(dev))
@@ -982,14 +990,14 @@ static void fw_domain_init(struct drm_i915_private *dev_priv,
 	setup_timer(&d->timer, intel_uncore_fw_release_timer, (unsigned long)d);
 
 	dev_priv->uncore.fw_domains |= (1 << domain_id);
+
+	fw_domain_reset(d);
 }
 
-void intel_uncore_init(struct drm_device *dev)
+static void intel_uncore_fw_domains_init(struct drm_device *dev)
 {
 	struct drm_i915_private *dev_priv = dev->dev_private;
 
-	__intel_uncore_early_sanitize(dev, false);
-
 	if (IS_GEN9(dev)) {
 		dev_priv->uncore.funcs.force_wake_get = fw_domains_get;
 		dev_priv->uncore.funcs.force_wake_put = fw_domains_put;
@@ -1035,8 +1043,13 @@ void intel_uncore_init(struct drm_device *dev)
 		dev_priv->uncore.funcs.force_wake_put =
 			fw_domains_put_with_fifo;
 
+		/* We need to init first for ECOBUS access and then
+		 * determine later if we want to reinit, in case of MT access is
+		 * not working
+		 */
 		fw_domain_init(dev_priv, FW_DOMAIN_ID_RENDER,
 			       FORCEWAKE_MT, FORCEWAKE_MT_ACK);
+
 		mutex_lock(&dev->struct_mutex);
 		fw_domains_get_with_thread_status(dev_priv, FORCEWAKE_ALL);
 		ecobus = __raw_i915_read32(dev_priv, ECOBUS);
@@ -1057,6 +1070,15 @@ void intel_uncore_init(struct drm_device *dev)
 		fw_domain_init(dev_priv, FW_DOMAIN_ID_RENDER,
 			       FORCEWAKE, FORCEWAKE_ACK);
 	}
+}
+
+void intel_uncore_init(struct drm_device *dev)
+{
+	struct drm_i915_private *dev_priv = dev->dev_private;
+
+	intel_uncore_ellc_detect(dev);
+	intel_uncore_fw_domains_init(dev);
+	__intel_uncore_early_sanitize(dev, false);
 
 	switch (INTEL_INFO(dev)->gen) {
 	default:
-- 
cgit v1.2.3


From 078595043b490a5cf0c272fe90151be9a5d48973 Mon Sep 17 00:00:00 2001
From: Mika Kuoppala <mika.kuoppala@linux.intel.com>
Date: Wed, 28 Jan 2015 15:25:05 +0200
Subject: drm/i915: Don't do posting reads on getting forcewake
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

The checking for ack and also any subsequent mmio access
will serialize with setting the forcewake bit. Drop the
posting read as superfluous.

Note that in the put side we still want to keep the posting read
as it will ensure that the hw sees our forcewake release in a
timely manner and doesn't keep the hw powered up.

Comment from Chris:

On Wed, Jan 28, 2015 at 05:54:14PM +0200, Mika Kuoppala wrote:
> Ville Syrjälä <ville.syrjala@linux.intel.com> writes:
> > IIRC the posting read from same cache line actually fixed real bugs. So
> > I'm a bit worried about dropping them. But I suppose it's possible only
> > the _put side was important for those bugs.
>
> I found these:
>
> commit 6af2d180f82151cf3d58952e35a4f96e45bc453a
> Author: Daniel Vetter <daniel.vetter@ffwll.ch>
> Date:   Thu Jul 26 16:24:50 2012 +0200
>
>     drm/i915: fix forcewake related hangs on snb
>
> commit 8dee3eea3ccd3b6c00a8d3a08dd715d6adf737dd
> Author: Ben Widawsky <ben@bwidawsk.net>
> Date:   Sat Sep 1 22:59:50 2012 -0700
>
>     drm/i915: Never read FORCEWAKE
>
> https://bugs.freedesktop.org/show_bug.cgi?id=51738
> https://bugs.freedesktop.org/show_bug.cgi?id=52424
>
> The snb here seems to survive gem_dummy_reloc_loop and
> gem_ring_sync_loop in here with the get side posting removed.

Note that we kept the once associated with #52424, but judging by my
comments in #51738 the posting read is just a band aid anyway as a full
mb() itself was not adequate.

Suggested-by: Chris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: Mika Kuoppala <mika.kuoppala@intel.com>
Reviewed-by: Chris Wilson <chris@chris-wilson.co.uk>
[danvet: paste relevant review discussion in.]
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
---
 drivers/gpu/drm/i915/intel_uncore.c | 1 -
 1 file changed, 1 deletion(-)

(limited to 'drivers/gpu/drm/i915/intel_uncore.c')

diff --git a/drivers/gpu/drm/i915/intel_uncore.c b/drivers/gpu/drm/i915/intel_uncore.c
index be2c7fcf5f1f..76b60a3538b2 100644
--- a/drivers/gpu/drm/i915/intel_uncore.c
+++ b/drivers/gpu/drm/i915/intel_uncore.c
@@ -131,7 +131,6 @@ fw_domains_get(struct drm_i915_private *dev_priv, enum forcewake_domains fw_doma
 	for_each_fw_domain_mask(d, fw_domains, dev_priv, id) {
 		fw_domain_wait_ack_clear(d);
 		fw_domain_get(d);
-		fw_domain_posting_read(d);
 		fw_domain_wait_ack(d);
 	}
 }
-- 
cgit v1.2.3


From 3225b2f95dbb9981be9e2002e49cd8abf0d8d01a Mon Sep 17 00:00:00 2001
From: Mika Kuoppala <mika.kuoppala@linux.intel.com>
Date: Thu, 5 Feb 2015 17:45:42 +0200
Subject: drm/i915: Squelch overzealous uncore reset WARN_ON

We added this WARN_ON to guard against using uninitialized
forcewake domains. But forgot blissfully that not all
gens have forcewake domains in the first place.

v2: Move WARN_ON to fw_domains_init (Chris)

Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=88911
Tested-by: Ding Heng <hengx.ding@intel.com> (v1)
Signed-off-by: Mika Kuoppala <mika.kuoppala@intel.com>
Reviewed-by: Chris Wilson <chris@chris-wilson.co.uk>
[Jani: add comment above WARN_ON as suggested by Chris]
Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 drivers/gpu/drm/i915/intel_uncore.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

(limited to 'drivers/gpu/drm/i915/intel_uncore.c')

diff --git a/drivers/gpu/drm/i915/intel_uncore.c b/drivers/gpu/drm/i915/intel_uncore.c
index 76b60a3538b2..c47a3baa53d5 100644
--- a/drivers/gpu/drm/i915/intel_uncore.c
+++ b/drivers/gpu/drm/i915/intel_uncore.c
@@ -166,7 +166,8 @@ fw_domains_reset(struct drm_i915_private *dev_priv, enum forcewake_domains fw_do
 	struct intel_uncore_forcewake_domain *d;
 	enum forcewake_domain_id id;
 
-	WARN_ON(dev_priv->uncore.fw_domains == 0);
+	if (dev_priv->uncore.fw_domains == 0)
+		return;
 
 	for_each_fw_domain_mask(d, fw_domains, dev_priv, id)
 		fw_domain_reset(d);
@@ -997,6 +998,9 @@ static void intel_uncore_fw_domains_init(struct drm_device *dev)
 {
 	struct drm_i915_private *dev_priv = dev->dev_private;
 
+	if (INTEL_INFO(dev_priv->dev)->gen <= 5)
+		return;
+
 	if (IS_GEN9(dev)) {
 		dev_priv->uncore.funcs.force_wake_get = fw_domains_get;
 		dev_priv->uncore.funcs.force_wake_put = fw_domains_put;
@@ -1069,6 +1073,9 @@ static void intel_uncore_fw_domains_init(struct drm_device *dev)
 		fw_domain_init(dev_priv, FW_DOMAIN_ID_RENDER,
 			       FORCEWAKE, FORCEWAKE_ACK);
 	}
+
+	/* All future platforms are expected to require complex power gating */
+	WARN_ON(dev_priv->uncore.fw_domains == 0);
 }
 
 void intel_uncore_init(struct drm_device *dev)
-- 
cgit v1.2.3